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Summary 

The experiments involved 11,571 abstracts (with titles), 1,000 key- 
word sterna and 93 search requests. Measures of word association are derived 
in several ways from the numbers of documents in which two given words 
co-occur, and measures of similarity from the numbers of words aissociated 
with botho Word clusters with different degrees of overlap are derived from 
the resulting networks of word connections for use as document descriptors. 
All are employed in retrieval and their perf orrianoe analysed. 
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Two new measures, sensitivity and coverage, reflect the variation in a 
strategy's performance from request to request. The best strategy depends 
on the user's requirements. For a single strategy key-words are simplest 
but the quantities of output are erratic and may usefully be controlled 
according to word associations. If two strategies can be used key-words 
alone may be followed by associations, yielding in a similar output quantity 
30 ^ more relevant documents. The corresponding use of clusters is marginally 
better but unlikely to justify its extra cost. 
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P<i53 VII. 4 after line 12 insert 

We have two versions of tlie harvest file on which document 
assessments are stored. On bhe standard harvest file HKG5^ 
there are 2020 known relevant documents, on the full harvest 
file HAROO there are 209O, 

□n HKG5^ the output corresponds strictly to the K values 
yielding K^!^50 for the 93 requests. 

□n HAROO assessments from all sources were retained in case 
the abstracts should be retrieved again in new runs, or in old 
runs at higher K'. 

These extra sources were: Runs 6,9.13,20 at K'ce51 ,52,51 ,52 
respectively (5 re levant Run 26 (42)j Rum 27 (24), the latter- 

pair having 1 In common. The K' mentioned come from the slightly 
higher K, namely 5^, 5^, 72, 7^, needed to give at a time when 

requests 1 ,6,46,66 had not yet been cancelled. In the other 
standard rrins K' was not altered within the nearest Integer. 

The requests benefited somewhat unevenly, but the net result 
Is that both overall and average known recall figures derived 
from HAROO are lower by about 3'l%* 

In the report Appendix b8 Is derived from HAROO; all other 
known recalls as on pp .53 , 5^ , Appendix B3 (v ) ,Appendlx B9b are 
related to the standard HKG50. 

Tables similar to Appendix B8 based on HKG5^ are available 
for all rijins except 24,26,27 up to 50 . 

Our main use of recall Is to obtain the estimates of 80% 
on p.5^: It does not enter directly Into our evaluation and Is 
tabulated for rough comparison with other experiments. 



after Appendix B9a Insert 

APPENDIX B9b 

93 Requests: Average Output 



Overall % Overa 11 % 

Run Relevant Irrelevant Total Precision Known Recall 



KWS 


13 


991 


3637 


4628 


21 .41 


49.06 


AWKWS 


22 


898 


3720 


4618 


19.45 


44.46 


ARM 


16 


782 


3835 


4617 


16.94 


38.71 


MCSOl 


14 


873 


3870 


4743 


18.41 


43.22 


MCS1 1 


20 


857 


3667 


4524 


18.94 


42.43 


RJR 


19 


939 


3764 


4703 


19.97 


46.49 


ARMSR 


17 


785 


3877 


4662 


16.84 


38.86 


SR14 


6 


753 


3852 


4605 


16.35 


37.28 


PDR14 


11 


861 


3825 


4686 


18.37 


42.62 


EAG3 


15 


828 


3811 


^639 


17.35 


40.99 


EAG4 


18 


835 


3764 


4599 


18.(6 


4l .34 


EARG4 


23 


914 


3793 


4707 


19.42 


45.25 


13T14 


9 


966 


3729 


4695 


20.58 


47.82 


13W14 


28 


991 


"^3689 


4680 


21 .34 


49.06 


Ul4 


21 


902 


3759 


4661 


19»35 


44.65 



^includes 36 not assessed 




The figure used for total known recall Is 2020 relevant 
(c.f. VII. 4. 111). 
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Corr Igenda 

p« 54 lines 3-6 read 

Request 10 11 53 

38 4 1 

91 9 5 

24 59 

Appendix b 4 second page line 3 
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PART 1 : Statement of Problem and Outline of Approach 



101 Scope of the 3N1PL experiment 

During the last two decades there has been an increasing awareness of 
the inability of fixed classifications, such as the Universal Decimal 
Classification (UDC), to cope with Document Retrieval, that is the provision 
of a list of references in response to a request for information on a 
specified sub,ject o At the same time, methods of indexing and retrieval based 
more directly upon the vocabul ary of documerts and requests have grown in 
popularity and there has been a marked trend towards the use of such systems 
by many of the more progressive technical libraries o 

Document, Reference , or Text Retrieval must be distinguished from Fact 
or Data Retrieval o A request for articles on 

'Radio Waves from the planet Jupiter* 
differs in kind from the queries 

'Which companies worth over a million 
pounds attempted takeovers in 1968?* 

'What is the Specific Heat of Copper? * 

In this report, we are concerned only ;vith the former o 

The work reported here, whicn began at NPL in 1 961 and was recently 
brought to an end, has been concerned with the improvement of key -word 
techniques in purely mechanical indexing and retrieval sys terns o Its 
objectives were 

(i) to develop methods of clustering words on the basis of 
specially computed statistical measures of association 
between word pairs, and 

(ii) to explore and evaluate ways of employing these clusters 
and associations to improve performance, especially in 
the ability to recall relevant material o 

We assume that natural language texts are 5n machine-readable formo 

102 The retrieval problem 

Uiven a collection of documents and a request on some specified topic, 
the problem n s to chose criteria which will select the documents likely to 
answer the request; will select them singly, or in groups, in order or 
likelihood; and will optimise the system performance as judged by the 
userso 



Thus any retrieval system must 

(i) subdivide the collection, preferably into mutually 
exclusive subsets, and 

(ii) order these subsets of documents for presentation to the 
requestoTo 
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I o 3 Comparison of fixed classification schemes and key-word based systems 

The view of the retrieval process taken above clarifies some of the 
main distinctions between hierarchical (tree-like) classification schemes, 
such as the UDC, and key -word based retrieval systems « 

In the case of a hierarchical classification scheme 

(i) The system provides a single, fixed, subdivision jf the 

document collection and a fixed ordering of the documents o 

(ii) The system itself does not specify the order in which the 
document or document subsets should be considered when a 
search is conductedo The requestor must decide this 
himselfo Cross-references and *see also^ entries are all 
the system provides to aid him in accomplishing this 
difficult tasko 

In contrast, in the case of key-word based systems :- 

(i) A special subdivision of the document collection is provided 
for each search requesto 

(ii) The document subsets are ordered according to some criteria 
for assessing the likelihood that a docziment answers the 
request. In the simplest case, the criterion is taken as 
the number of key-words in the request that were also assigned 
to a document to index ito The collection is t;hen subdivided 
so that all the documents in one subset score the same 
number of points and the subsets are ordered according to 
these scores. 

There is no reason to suppose that a unique arrangement of the 
document collection into subsets can be found that would be best, or even 
good, for all possible requests. In fact, this seems highly lonlikelyo 
Yet that is an assumption underlying any classification scheme like che 
UDCo The chief benefit of the UDC is to decide the arrangement of the books 
on the shelves. In this respect key -word based systems are superior in 
providing a tailor-made subdivision for each request. They also have the 
great advantage of producing an ordering of the dociiment subsets for 
presentation to the requestor. As will be seen later, some of the more 
elaborate key -word systems can produce a subdivision of the document collec- 
tion into a very large number of subsets, producing quite a fine ordering 
or partial ordering of the collection. This enables the requestor to 
terminate his search when he desires, giving him considerable control over 
the quantity of output he receives from the system. 

1.4 Advantages of mechanical systems based upon key-words 

(i) Being mechanical, they do not suffer from human indexing 

inconsistencies, as do the more traditional systems such as 
the UDC. 

(ii) The key -words may be combined as and when necessary in order 
to describe the subject content of, or to index, a document, 
or formulate a request for information. Q?hus, a document 
describing a piece of electronic equipment used for measuring 
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the human heart-rate could have been indexed by such words 
as ’medical*, ’heart’, ’rate’, ’electronics’, ’instrument’, 
’measurement’, even before the development of the ’medical 
electronics’ field had been anticipated, and even if those 
words had never before been brought together within the 
systemo By contrast the UDC, for example, sets out to 
anticipate all subject areas of interesto Unpredicted areas 
covering more than one discipline cannot be accommodated 
without considerable, and in practice continual, revision of 
the scheme o 

(iii) Being mechanical and therefore amenable to computer handling, 
they boast the advantages associated with modern computing 
machineso These are, principally, high speed and reliability 
and comparative «rt,se of repeatability. At present texts are 
key-punched by hand from the printed copy, but more and more 
material is being put into machine-readable form even before 
printing o 

(iv) The processing speeds possible with computers make it 

practicable to consider procedures of a complexity that would 
otherwise render them quite unfeasible o This is a major 
advantage of such systems, and it creates a vast range of 
potential modes of operation still largely unexploredo 

Without the aid of computers it would not be sensible to consider the 
techniques described in this report, as the manual processing involved in 
the word association and word clustering stages would be quite prohibitiveo 

I o 5 Systems based upon simple comparison of key words 

The simplest systems operate by comparing a set of key -words used in 
formulating a search request with sets used to index document o In the 
collectiono There are two common ways of choosing k^-words when indexing: 
derived indexing, ir^ which words are selected from the title or abstract 
and, perhaps, from the full document text, and assignment indexing, in 
which the words used to index a document are decided by other means, most 
often according to the intuitive judgement uf human indexer So The usual 
practice is to retrieve first any documents indexed by at least all of the 
words used in formulating the requesto If insufficient output is produced, 
or if the searcher thinks that not all relevant material has been gathered 
in this way, it is customary to relax the request formulation and to 
retrieve anything indexed by at least all but one of the request wordSo If 
the request consists of more than two words it can be further relaxed in 
this way to obtain still more output o A slightly more elaborate procedure 
allows the request to be formulated as a Boolean function (involving logical 
connectives such as AND, OR, NOT) of a set of key -words o 

Control of the indexing language sometimes involves no more than 
recognition of synonyms and perhaps the use of some form of ’user’s 
dictionary’ or ’scope notes’ giving guidance on the choice of words in an 
effort to achieve a modicum of indexing consistencyo Key-word stems may be 
used in place of key words, especially when they are chosen from the text 
itselfo This means conflating or identifying morphological variants such 
as ’act’^ ’acts’, ’actor’, ’actod*, ’active’o This can be done mechanically 
to a satisfactory degree, for example by matching with as long a stem as 
possible in the machine-held dictionary (See IIo9), 
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Io6 The need to search under related terms 



The richness and flexibility of natural languages are such that almost 
any search request can he expressed in numerous alternative waySo Different 
people will have different ways of asking for essentially the same thing 
and, indeed, any individual might express the same request in different ways 
on different occasionso For example, one would he asking for virtually the 
same thing hy requesting information on 

'transistor phase-splitting circuits' 



and on 



'driving circuits for transistor push-pull amplifier So ' 

A simple system of the type discussed in the previous section would not 
recognize this fact and, finding that the requests apparently have little 
in common, would process them accordin^yo 

Another factor is that very few requests are such that all relevant 
documents would he retrieved if attention were restricted to those hearing 
a close ■oarallel to the form of the requesto Take Bar-Hillel's example of a 
search request for documents dealing with 

'diseases of animals in South Americao * 

One can envisage documents with such titles as 

'Bacteria living in dogs' 



and 



'The life cycle of insect X' 

which may well he relevant and would certainly he worthy of considerationo 

This is not a mc4,tter of allowing for alternative expressions of the 
same idea, hut one of extending the search to cover subject areas related to 
that of the request.* Once again the recall of relevant material should he 
improved hy searchin.g under terms related to those in the requesto 

Io7 Semantic relationships 



Many kinds of semantic relationship can exist between words or phrases 
whether within one text or in two comparable textso Such relations include 
synonymy, morphological variation, attribution, cause-effect, part - whole; 
more broadly, similarity of function, even physical proximity where the 
connection is frequent enough© Almost any association can attach itself to 
the meanings of words over a period of time, or conversely, may decay or 
vanisho The strength of a semantic association may also vary from library to 
library o Some associations apply only within particular documents; others 
to a subject area of the particular lihraiy, or to the library as a whole; 
others more widely stillo The indexing procedure and/or the search strategy 
should take this into accounto Thus with the example given above, documents 
should be considered for retrieval if indexed by such words as 'bacteria' 
and 'insect' o 
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Under such circumstances, the retrieval strategy must decide the 
relative importance of documents indexed by request words, by semantically 
related words and by combinations of botho 

Io8 Structural relationships 

The underlying structure of a text can be described in terms of 
relationships between words or other components of the texto Any transitive 
verb, in so far as it indicates some specific sort of connection between a 
subject and an object, constitutes a structural relationshipo All the 
classical syntactic relationships are structural in nature. Structural 
relationships are exhibited, for example, by the qualification of a verb or 
adjective by an adverb, or of a noun by an adjective. Again, in describing 
the content of technical doc\aments, the ’effect (or action) of A upon B’ 
might be used as a structural relationship. In this case A and B might be 
chemical elements or compounds, machine components, processes, environmental 
conditions, etc. 

Consider the following title: 

Effect of transverse field on switching rates of magnetic-core 

storage systems. 

Examining some of the syntactic relationships we note that ’storage* 
qualifies ’system*, ’magnetic’ qualifies ’core’, ’magnetic-core’ qualifies 
•storage system’, ’switching rates’ pertains to ’magnetic- core storage 
system’, ’transverse’ qualifies ’field’, etc. 

These structural relationships, unlike semantic ones, have no existence 
outside the texts in which they occur; they are properties of particular 
texts rather than properties of large corpora. 

In the context of document retrieval systems it is still far from clear 
what use, if any, can be made of structural relationships existing in the 
documents or in requests. Is it helpful to preserve the structural information 
as far as possible when indexing? This is done in the SYOTOL system [lj 
and also in the intricate relational indexing system developed by Parradane 
[2]o On a much lower level many systems employ links to indicate the 
existence of relationships at the indexing stage: words, phrases, etc., 

that are interrelated are tagged and pairs of common tags indicate entities 
which are related, without defining the sort of relationship. 

If structural relationships are specified when indexing documpncs and 
only those documents retrieved which show a sufficiently good correspondence 
with the structure of a request (it being automatically assigned that the 
words used to index the retrieved dociaments correspond satisfactorily with 
those of the request), then the chances are very high that documents of 
likely interest will be missed. Once a request is transformed or matched 
against a document using semantically related words, the structure is likely 
to lose its parallel, particularly, of course, if the matching words are 
spread over several sentences in the document. 

1.9 What use should be made of relational information for document retrieval? 



Prom the discussion in the previous section there seems little case for 
the introduction of structural or syntactic relationships. On the whole, such 
information is far too specific in nature to be useful if one is concerned in 
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improving the recall of relevant documentSo If one’s prime interest were in 
minimizing the recall of irrelevant documents, or if one were setting up a 
fact-retrieval system, different considerations would apply© 

On the semantic side synonymy and near- synonymy are not the only things 
of interest, all other relationships are potentially useful as pointers to 
words, additional to those used in a request, that should be considered in 
the retrieval strategy© Our prime concern is to improve retrieval systems 
from the point of view of their Fbility to recall as much relevant material 
as possible© That being so, the most valuable information is that specifying 
which words are semantically related© There is little further advantage in 
distinguishing between the various kinds of relationship occurring© 

Cor =?ration shows that the necessary information about semantically 
related \ is is not readily available© The task of investigating and 
recording all the relationships for a large technical vocabulary is enormous© 
Furthermore, the job would have to be repeated from time to time because the ' 

situation is dynamic, new relationships being formed and old ones dying out 
as subjects develop© To some extent the relationships existing depend upon 
the interests and subject coverage of each particular library© For these 
reasons, and in the interests of greater objectivity, it would be a tremendous 
advantage if the identification of semantic relationships could be mechanized© 

A possible method of doing this, which has been tried experimentally, is discusse 
in the next section© 

I©1 0 Statistical word association 

Suppose that a large set of texts is analysed statistically, observations 
being made of the total number of distinct words, of the number of sentences 
containing each word and the number of sentences containing each possible 
pair of distinct words© From a knowledge of the number of sentences in the 
full set of texts and of the number of sentences containing each word, and 
making the assumption that words are statistically independent in their 
occurrence in texts, it is simple to calculate the expected number of sentences 
containing any given pair of words© If the number of sentences in which two 
words are actually^ob served to occur does not differ too greatly from the 
calculated expected number, then it is reasonable to suppose that the assumed 
statistical independence of the given words is verified© If, on the other 
hand, these observed and expected quantities are significantly different, then 
the assumption has not been verified and the words in question must be assumed 
to be statistically associated© Techniques exist for deciding what is 
significant in this respect and for obtaining a quantitative measure of the 
degree of statistical association between two words© 

Suppose the sample of texts is so large that the measured statistical 
associations may be taken as representing properties of the language, rather 
than peculiarities of the particular sample of texts (at least within a 
given subject area if not in general)© It is then reasonable to hypothesize 
some, albeit unidentified, semantic relationship whenever a significant 
statistical association is found© The measure of association being 
statistical, there is always a chance that a high association that looks very 
significant has, in fact, arisen fortuitously© Steps must be taken to control 
this© These considerations are fully dealt with in section III, where we ^ 

define and measure an association factor for any pair of words© We cannot 
assume directly that, because two words frequently co-occur within texts , 
therefore one is a good substitute for the other© 




12 



- 7 - 



Accordingly, for retrieval purposes, we develop a similarity coefficient 
derived, again mechanically, from the association factoro This measures the 
tendency of two given words to be found in similar company o 

In11 Word clusters 



Having established a network of statistical connections between a set 
of words by a technique of the sort just described, it is interesting to 
investigate whether there is a tendency, by virtue of these interconnections, 
for the words to cluster . Assuming fairly restrictive rules of cluster 
formation, so that the degree of interconnection within clusters is relatively 
high, identification of such clusters offers the following advantages:- 

(i) A useful relationship for retrieval purposes may be hypothesized 
between any pair of words in a cluster o This provides a way of 
predicting useful connections between words in many cases where 
they do not result directly from the initial statistical analysiso 

(ii) Instead of indexing documents in terms of individual words or 

phrases, each cluster of closely associated words could be used 
as a descriptor**' to be assigned to documents to index them and 
to be used for formulating requestso This method of utilizing 
the word associations, if effective, is economical in terms of 
storage space required for indexing a large collectiono Further 
economy is achieved by not having to store explicitly all the 
separate word associationso 

Io1 2 Scale and subject area of experiments 

A collection of some 12,000 abstracts of papers in electronics, 
computers, physics and geophysics was used as the corpus for the statistical 
analysis and derivation of word associationSo The same collection formed 
the document base for later retrieva:. runs in which various indexing and 
search strategies and several different sets of word clusters were testedo 
A dictionary containing 1,000 key -word stems was compiled on the basis of 
a sample of the abstracts© These are the only words, taken separately or 
in clustei's, used in the experiments for indexing and searching© 

Each retrieval run involved searches for 93 requests, 7 of an original 
set of 1 00 having been cancelledo A variety of strategies involving the use 
of five different sets of word clusters, and some strategies employing the 
statistical associations directly, no clusters being involved, have been 
tested and compared in a total of 14 main retrieval runs© 

I©13 Relevance and coordination 



All evaluation of performance is based on subjective considerations of 
the relevance of retrieved material by the 20 people who supplied the search 
requests, each assessing his own as relevant or not© Within this report we 
use the word relevance to refer to this subjective judgment by the user, 
and not to any decision by the machine© The machine as we have said earlier 



*By descriptor is meant any key-word, phrase, word cluster, decimal 
classification number, author *s name or any other entiiy assigned to 
documents for indexing purposes, or used to formulate search requests. 
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(lo2) puts the documents into subsets for presentation in a particular ordero 
Each strategy has its own scoring system for this purpose* The score measures 
in some definite way the extent to which the key-words in an abstract match 
those in the request, and we refer to it as the coordination o Then each 
subset contains all documents with a given coordination, whether relevant 
or noto The machine has to decide how many subsets to output, beginning with 
the highest coordination, and for this it requires further infonnation from 
the user. 

1014 Output and evaluation 

A customer with a request must specify approximately the size of output 
he requires, for various reasons* First, there is likely to be a limit on 
the number of documents he is willing to scan* Secondly, the cost of an 
operating system is closely related to the quantity of output [ 3 J ° Other 
methods of specification such as a desired coordination level are likely to 
occasion very wide and therefore costly or inconvenient variations* It is 
not easy to foretell, for example, the effect of asking for all documents 
with a particular number of key -words present* Accordingly, strategies are 
evaluated by comparing the numbers of relevant documents in a given size of 
output, and also by the relevant documents retrieved by one strategy and not 
by another* 

1 01 5 Recall 



In experiments where the whole library has been evaluated, that is every 
document compared with every request, it has been common practice to 
generate output for each request till a certain proportion of its relevant 
documents have been retrieved (recall ratio) and then to measure what pro- 
portion these are of the output (precision ratio) o One chief reason for 
doing this has been to allow for requests with few and with many dociaments 
in the collection* It does not however correspond very closely to the actual 
business of running a system, where the relevant docioments are not known 
beforehando In our own experiment this would require 11,571 times 93? say 
one million relevance assessments, which was beyond our resources* In fact 
about 17?000 were made by our 20 requestors* To estimate the actual, n-umbers 
of relevant documents by sampling from part only of the library woul.d have 
been ineffective, since the average is only 21 o7 per request* However, we 
felt that with 14 different strategies we were likely to cover most of the 
relevant documents* An almost exhaustive search with 3 requests and another 
using subject indexes with 12 requests suggested that in far ^ we had already 
found about 8C^ of the relevant documents in the collection* We tabulate a 
known recall ratio corresponding to the numbers of known relevant in a given 
output* 
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PMT 2 ; Word Association and Clustering 
II Preparation and Machine Input of Texts 

1101 Choice of texts and sub,ject matter 

The principal use of the texts in the first phase is word and word- 
pair count ingo If the statistical associations produced by analysing the 
texts are to be as meaningful and useful as possible, texts should be used 
which do not contain a high proportion of repetition or redundant ’padding* 
material o Abstracts have the desired characteristics and that was the form 
of text selected for these experimentso 

The abstracts were compiled at the Radio Research Station (now Radio 
and Space Research Station) at Slough, England o They were published over 
the period 1953-62 in a journal which, during that time, has been called 
’Wireless Engineer * , ’Electronic and Radio Engineer* and ’Electronic 
Technology* o The same abstracts were published concurrently in New York in 
the Proceedings of the Institute of Radio Engineers (now the Proceedings of 
the Institute of Electrical and Electronics Engineers) o 

Out of some twenty categories under which these abstracts appeared the 
following five were selected for our experiments: 

1 o Automatic computers 

2o Circuits and circuit elements 

3o General physics 

4o Geophysical and extraterrestrial phenomena 

5o Subsidiary apparatus o 

These subjects were chosen partly because we have experts at NPL in 
electronics and computers who could assist uso At the same time it was 
thought that, from the retrieval point-of-view, these subject areas would 
provide most of the problems and difficulties encountered in other scientific 
and technical areaso Geophysical material was incH uded so that the collection 
should not be too narrow in scopeo We have been very grateful for the 
willing cooperation of the Radio and Space Research Station in providing us with 
the advice and assistance of subject experts in this fieldo 

It should be pointed out that the scope of the abstracts is not as 
wide as the category headings would suggest o Abstracts appear under these 
headings only for papers having some pertinance to radio-communications o 

1102 Unit of text 



In studying word co-occurrence in texts for the purpose of computing 
statistical association measures, what should constitute co-occurrence? 
Juxtaposition of two words mi^t he taken as the criteriono Alternatively, 
two words may be said to co-occur if they appear in a text within some 
broader, but well defined, context such as a span of a specified number of 
words, a sentence, a paragraph, a page or a complete paper o In any case it 
is necessary to work with some such unit of text and to define word 
co-occurrence in terms of ito The disadvantages of adopting too small a 
unit are that many useful associations might thereby be missed, and secondly 
that an inordinate amount of text must be analysedo For example, if word 
Juxtaposition were taken as the basis for oo-occu3?rence an astronomical 
quantity of text would have to be scanned in order to uncover a significant 
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proportion of useful associationso The main penalty for working with too 
large a unit of text is that there would then he a high chance that any 
associations produced would not he very meaningful o There is prohahly an 
optimum unit of text for any proposed method of analysiso A considerable 
amount of experimentation would he required to establish this optimunio 
However, the point was not thought to he sufficiently critical to justify 
this and the choice was made less ohjectivelyo 

Each title and abstract was regarded as a unit of text, these units 
containing on average a total of about 33 wordso Repeated co-occurrence of 
two words in abstracts of papers on fairly well defined topics should be a 
reasonable indicator of the existence of a useful relationship between them© 
Many of the fortuitous relationships found in this way should be eliminated 
by rejecting the weaker statistical associationso 

1103 Selection of key-words 

This is based upon a study of a sample (taken from the corpus) of 1 g6k£> 
abstractso A listing was produced of all the distinct words occurring in this 
sample together with the frequency of occurrence of (i.eo the number of 
abstracts containing) eacho This list was studied very carefully by three 
people, two of them being fairly familiar with the subject matter, who decided 
intuitively which words to retain in the system as k^-words, all others 
being excluded from further considerationo 

1104 Conflation of word forms 



Where it was thought to be appropriate no distinction was made 
between different form of the same word. For example, computer, computers, 
compute, computing, computable, computation, etc#, are not distinguished. 
These forms are conflated by identifying all words starting with the stem 
COMPUTo This is consistent with the view expressed earlier that structural 
details are relatively unimportant for document retrieval systems, the 
information discarded by conflation being mainly syntactic. 

Conflation also has the following advantages: 

(i) It reduces the size of the dictionary (of stems) to be stored 
and manipulated in the computer, and 

(ii) When computing statistical association measures between word 

stems the frequencies and co-occurrence frequencies involved will 
be greater, for a given corpus, than those pertaining to the 
separate word forms# This enables us to work with a smaller 
corpus. 

Another way of thinking about this is that, from the point-of-view of 
indexing documents and their comparison with requests, it is important that 
the concept of ’computation* be discussed in a document or introduced in an 
abstract, but it seems of relatively little interest to know whether the 
noun or verb is used or whether machines are discussed in singular or plural© 

IIo5 The stem dictionary 

The system of word stems was constructed so as to maintain those 
distinctions between words thought to be useful for the purpose of these 
experiments and to suppress others# As an example, the stem ANALY appears 
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in the dictionary, but ANALYSER is also entered as it was decided that it 
might be useful to be able to distinguish such references to pieces of 
hardware (differential or spectral analysers) from other occurrences of the 
shorter stemo In a few cases word forms that would have been conflated 
have not been, because to do so simply by si: ortening the stem would permit 
undesired words to match the same stemo The stem CIRCL matches with circle, 
circling, etco, but had the stem been shortened to CIRC in order to match 
also with circular an undesired match would occur with the word circuit o 
The dictionary therefore contains CIRCULAR as a separate entryo 

Having conflated terms in this way all the stems were listed and their 
frequency of occurrence within the 1 ,6A8 abstract sample v/as recordedp At 
this stage there were 1 ,324 stemso Of these, 323 occurred in the sample 
once only and 171 occurred twice o It was decided that most of the words 
occurring in the sample only once were of little value in the experiment and 
should be discardedo However, fourteen of them v^ere retained as being useful 
words within the subject area in spite of their single occurrence in the 
sample o 

On consideration of possible methods of organizing and handling the 
dictionary and other data in the computer it was found that if the number of 
items in the dictionary could be limited to 1,000 much more efficient use 
could be made of the machine and programs v^ould be executed more rapidly than 
would be the case with a larger dictionaryo The decision was therefore taken to 
discard a further fifteen of the rarer words, leaving exactly 1,000 key-word 
stems in the final dictionaryo I'lie complete dictionary is shown in 
Appendix Alo 

IIo6 Word truncation 



It was observed during construction of the dictionary that practically 
all the distinctions required to be made between words could be made on the 
basis of the initial eight letters, or fewer, of each wordo Again in the 
interests of economy, on being read into the computer all text words of 
greater length are truncated after the inihial eight letterso The very small 
number of distinctions which consequently cannot be made seems a small 
sacrifice for the computer storage space savedo This is purely a matter of 
convenience, and although the loss of performance incurred is acceptable for 
the purpose of our experiments it might not be considered so if the design 
of an operational system were being consideredo 

IIo7 Key-punching of texts 

A typical abstract from the corpus is shown in Figo IIo7o1a The title 
and full text of each abstract are coded for machine input, with the 
following exclusions: 

1) abbreviated bibliographic details (ioe. author, journal, etco) 

2) numerical data, 

3) special symbols, 

4) all p^unctuation marks other than the full stop, 

5) full stops in abbreviations o 
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551.510.535 2052 

Sequential and Lunar Effects on the Equatorial 

E^.-’-S. Matsushita (/. Geomag. Geoelect. ^ Sept. 1955, 
Vol. 7, No. 3, pp. 91-95.) Among the various i':ypes of 
E., which have been observed, one shows apparent 
vertical movement on the ionogram, and has been 
termed 'sequential E/ by investigators at the National 
Bureau of Standards. A study is made of this pheno- 
menon using records from a number of station^:; the sub- 
type investigated is that involving an E, region which 
first appears at a height of about 200 km in winter and 
180 krn in summer and then drops to normal E level, 
where it persists for scmie hours. 'J'he latitude and 
time distributions of the phenomenon are briefly 
discussed . 

Figo IIo7*1 o Sample abstract from corpus 



This is key-punched on to 80-column cards using a standard 4-z-one code 
(see FigoA2o1 5 Appendix A2)o The one spare code position is used to represent 
a full stopo The termination of an abstract is marked by two full stopso 
Each character or space occurring in the material being key-punched is coded 
in a separate column on a card, the first 48 columns only of each card being 
used for this purpose o The coding of each abstract commences on a new card 
and abstracts involving more than 48 characters are continued on a subsequent 
card or cardso Figo IIo7o2 shows the form in which the sample abstract is 
coded for machine input after the various exclusions have been madeo 

SEQUENTIAL E AND LUNAR EFFECTS ON THE EQUATORIAL Eo 
AMONG- OHE VARIOUS TYPES OF E TOICH HAVE BEEN OBSERVED 
ONE SHOWS APPARENT VERTICAL MOVEMENT ON THE lONOG-RM 
AND HAS BEEN TEEMED SEQUENTIAL E BY INVESTI&ATORS AT 
THE NATIONAL BUREAU OF STANDARD So A STUDY IS MADE OF 
THIS PHENOMENON USENFG- RECORDS FROM A NUMBER OF 
STATIONS THE SUBTYPE INVESTI&ATED IS THAT INVOLVING- AN 
E REG-ION WHICH FIRST APPEARS AT A HEI&HT OF ABOUT IN 
WINTER AND IN SUMMER AND THEN DROPS TO NORMAL E LEVEL 
WHERE IT PERSISTS FOR SOME HOURSo THE LATITUDE AND 
TIME DISTRIBUTIONS OF THE PHENOMENON ARE BRIEFLY 
DISCUSSEDo 



Figo IIo7*2o Sample abstract from corpus as key-punchedo 



The exclusion, during the k^-punching operation, of various parts of 
the text was for the purpose of simplifying the task and thereby reducing its 
cost, and associated error rateo This applied particularly to bibliographic 
datao Much of this work had to be contracted out, but it was very difficult 
to find people with experience in this kind of worko We therefore had to 
make the requirements as simple as possible in order to get it done at alio 

In spite of these precautions the key-punching took many months and 
the error rate was alarmingly high at first® In the first batch of about 
1 ,600 abstracts, key-punched by an outside agency, about one word in twenty 
five was incorrectly punchedo This material was prepared by girls operating 
ordinary hand punches, making two finger depressions to obtain the alpha-numeric 
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codingo Most of the remaining punching was done on machines having a type- 
writer style keyboardo Fortunately the error rate for the 10,000 or so 
abstracts following the original batch dropped to about one incorrect word 
in 25O0 Presumably this is at least partly attributable to the use of card 
punches with a full alphanumeric keyboardo 

The latter error rate of 1 in 250 was not expected to have a pronounced 
eff*ect upon the outcome of the statistical analyses and was therefore 
accoptableo The higher error rate in the first batch would not have been 
acceptable had it persistedo However, as it affected only a small proportion 
of the corpus those abstracts were also used without correctiono 

II 08 R epresentation of text in the computer store 

Stems in the dictionary and text words read from punched cards are 
stored in the computer in an extremely simple coded formo Each letter, and 
space, is represented by a five digit binary number: the nth letter of the 

alphabet by n, space by the number zero* The coded representations of 
successive characters of a word or piece of text, taken in their natural 
order, are held in successive groups of five binary digits in the computer 
store, progressing from the more signif leant to the less significant end 
of each computer word** To illustrate this consider the representation within 
the computer of the word ’machine’ 0 These characters are first transformed 
into a sequence of numbers representing the position of each character in the 
alphabet, thus 



13, 1, 3, 8 , 9 , 14, 5 » 

The binary representation in the computer is obtained simply by grouping 
together the corresponding five digit binary numbers: 

MACHINE 
01101 00001 00011 01000 01001 01110 00101 

T 

more significant 
end of computer word. 

The text of each abstract, when read into the computer from punched 
cards, is stored as a continuous character string, running on from one 
computer word to the next and occupying as many words of computer store as 
are necessary* The text words are then separated by identifying the space 
characters, which have their own particular code* After truncating long text 
words by discarding all letters after the eighth, each text word is stored 
in a separate computer word, each of the latter being filled from the more 
significant endo 



computer store is commonly composed of units referred to as ’words’, each 
able to hold a string of binary digits* Each word of the Ace Computer, 
used in these experiments, contains if8 digits* By the more significant end 
of a word is meant the end at which, if the string of digits stored in the 
word is regarded as a binary number, the most significant digit of that 
number is locatedo 
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IIe>9 Dictionary organization and look-up 

Since text words are trxmcated to a maximum of eight letters when they 
are read into the machine, no character string of greater length is ever 
compared with the stems In the dictionary during the look-up processo The 
longest stems in the dictionary therefore contain eight letters o The stems 
are stored one per computer word, coded in the same way as the text wordso 
The dictionary is arranged in eight sections, according to stem length, 
the stems in each section being arranged in alphabetical ordero 

The method of coding described requires 40 binary digits to represent 
the longest stems o Words in the Ace computer are 48 digits long, so that 
each computer word containing a dictionary stem has at least eight spare 
digits, some of these digits are used to store a number, in the range 1-8, 
associated with each stem indicating the longest text words to be considered 
when testing for a ^match* with that stemo By definition, a match cannot 
exist involving a text word of more than the specified maximum lengtho 
However^ since long text words are truncated after eight letters, when the 
number specified is eight, there is effectively no restriction on the length 
of text words with which the corresponding dictionary stem might matcho The 
number indicating the maximuni allowable extension of each stem is shown with 
the dictionary in Appendix A1 o 

To illustrate this system, consider the following dictionary entries 
(which are not adjacent): 



BAND 5 
BAKDWIDT 8o 

The first of these stems is intended to match with band and bands, no match 
with words of more than 5 letters being permit tedo The word bandwidth is 
picked up by the other stemo Although it does not arise in this case, the 
system as defined so far could result in the occasional mismatching of a 
text word with a stemo Consider the following dictionaiy entries: 

MODEL 6 
MODE 5o 

Since the stem MODE can be matched with words having 5 letters a mismatch 
could arise between this stem and the word modelo However^ this is avoided 
by accepting the longest stem with which a given text word matches o The 
dictionary being in sections of different stem length, this simply entails 
referring each word first to the section containing the longest stems, then 
to the others in order of stem length, and always accepting the first 
match for each wordo 

The text of an abstract having been read into the computer, before its 
words are referred to the dictionary they are alphabetically ordered and 
multiple occurrences of words are eliminatedo The words in each section 
of the dictionary are also in alphabetical order, so all the words from the 
abstract are compared with. the stems, and matches noted, in a single pass 
of the dictionaryo The dictionary is stored on magnetic drums, so this 
procedure saves a great deal of time in terms of drum accessingo 

Corresponding to each abstract processed in this way a post-abstract 
is output on punched cards containing the code numbers of all dictionary 
stems occurring in the abstracto Other data output at the same time include 
the total number of words (including repetitions and words not in the dictionary) 
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axid the number of distinct dictionary stems in the abstracto 
IIo1 0 Word/abstract incidence matrix 

This is a binary matrix (ioeo a matrix whose elements all have value 
either 0 or 1 ) having a row corresponding to each dictionary word stem and 
a column corresponding to each abstract ^Figo Ilo10ol)o 

ABSTRACT 




(VALUES fictitious) 

Figo II 0 IO 0 I 0 Word-abstract matrix 



The complete matrix is far too large to be held in the machine, even on the 
magnetic drums, in its entirei^o It is therefore constructed in sections, 
each having 1 ,000 rows in correspondence with the word stems and 1,536 
columns, and therefore representing only a fraction of the collection of 
abstract So 

Before constructing a section of the incidence matrix all the elements 
are set to zeroo The post-abstracts are then read into the computer in 
sequence o As each one is read in the rows of the matrix representing the 
word stems listed in the post-abstract are rec-d from the drums, the element 
in each rcw corresponding to the particular abstract is set to unity and then 
the matrix rows thus modified are written back on the drumso After each 
section of the word/ abstract incidence matrix is completed it is output on 
punched cardso 
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III Derivation of Statistical Associations 
III.1 Scale of experiment What corpus size? 

There are two main considerations in deciding the scale of experiments 
of this kindo The firf-t is that it should be large enough, for the experiment 
to be realistico The second is that the corpus should be sufficiently large 
to yield statistically significant resultso 

Such an experiment should be regarded as being realistic if its results 
and conclusions can be extrapolated and applied to the solution of problems 
arising in real-life retrieval situations o Some of the problems encountered 
in practice arise through semantic ambiguity, the interpretation of words 
having different meanings or connotations in different contexts and through 
the vague and imprecise use of wordso In any experiments, therefore, the 
vocabulary should be sufficiently large and the subject coverage of the 
collection sufficiently diverse to ensux’e that factors such as these produce 
significant problems to be contended with. Results would be worthless if such 
difficulties were eliminated at the outset by oversimplification in setting 
up an experiment. 

How many documents should a system provide in response to a customer's 
request? This question has no simple answer. It will depend upon the nature 
of the request, the nature of the customer *s interest in the subject and 
upon many imponderable factors. However, most customers will have only 
limited time to consider the output from any system. Hence, given a vast 
collection and a system capable of turning up more and more relevant 
documents the limiting factor remains the maximum number of documents the 
enquirer thinks it reasonable to receive. In other Arords, the larger the 
document collection the more discriminating power required of the system. 

This is another disadvantage of experimenting with an unrealistically small 
^-ollection. 

Turning now to the subject of statistical significance, it should be 
noted that nothing can be inferred from a single co-oocurrence of two words, 
and tv/o may mean very little, regardless of the collection size. (As 
explained, the basic unit in these experiments is the word-stem, and *word% 
used here for brevity, should be interpreted in this way). It is therefore 
important to ensure that the outcome of an experiment is not critically 
dependent upon associations derived from such very small co-oocurrence 
frequencies, many of which may be purely fortuitous. In order to obtain 
larger co-occurrence frequencies in a reasonable number of cases most of the 
words should appear in the collection at least a dozen or so times. Rarer 
words are unlikely to yield the hi^er co-occurrence frequencies desired. 

An analysis of our corpus* shows that: 

1C^ of the dictionary stems occur in not more than 1 in 750 abstracts, 

2^0 of the dictionary stems occur in not more than 1 in 400 abstracts, 

and 

G6fo of the dictionary stems occur in not more than 1 in 100 abstracts. 
These figures make it clear that, for the given subject coverage and 



^Individual word frequencies are included in l;he listing of the dictionary 
in Appendix A1 . Distributions of word frequencies, for the full corpus and 
for different sized subsets, are shown in Appendix A4o 
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’vocabulary, the corpus must contain at least about 5 >000 abstractso With 
fewer abstracts than this many of the rarer words would yield no useful 
associations and would effectively be excluded from the experimento The 
vocabulary, which must already be regarded as being of minimal size for a 
realistic experiment, would thereby be made still small ero 

Regarding the figure of 5 >000 as being borderline, it was decided that 
the corpus for these experiments should contain a minimum of 10,000 abstractso 
A total of 12,288 abstracts were finally used for the derivation of word 
associations and clusters, although a few of these were excluded from the 
document base used subsequently for retrieval purposeso 

When this work began the comparable experiments of which we were aware 
involved very smeJ.l numbers of requests and only a few hundreds of documents o 
(Compare [20j , pp.104> 178)o With perhaps one or twc exceptions this 
situation has not altered (1969)0 

IIIo2 Variation of word co-occurrence frequency with corpus size 

For convenience of handling, the abstracts were processed in batches of 
1 >536, that being equal to the nvunber of bits (binary digits) in one 32-word 
block of memory in the ACE computer* The full corpus is composed of eight 
such batches o Pistributions of word co-occurrence frequencies appear in 
Appendix A4.0 These are for subsets of the corpus consisting of one, two and 
four batches, and for .-11 eight batcheso The cumulative totals shown in 
Figo IIIo2o1 are taken from these distributions. They f.ndicate the number of 
distinct word pairs with frequency greater than each given value, excluding 
self-pairings. The number of such pairs possible with a 1,000 word dictionary 
is 499> 50O0 




WORD 

PAIR 

FREQUENCY 


NOo OF BATCHES OF 1,53^ ABSTRACTS 


1 


2 


4 


8 


> 0 


56,379 


89,434 


1^,346 


202,721 


> 1 


17,601 


36,078 


73,392 


1 24, 009 


> 2 


8,276 


19,781 


46,251 


87,534 


> 5 


2,092 


6,522 


19,167 


43,911 


> 10 


532 


2,174 


7,882 


21 ,824 


> 20 


98 


552 


2,610 


12,698 


> 50 


1 


50 


404 


2,100 


>200 




1 


4 


98 


>500 
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Figo III.2o1 Number of word pairs with given frequencyo 
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Of all the possible word pairs, roughly 10^ occur in the single batch of 
abstracts analysed, rising to about 4C^ in the full corpuso The number of 
word pairs occurring with higher frequencies is of sp cial interest and it is 
seen that, whereas only 98 word pairs with frequency greater than 20 occur an 
the single batch, the corresponding number rises dramatically to 12,698 in 
the full corpuSo Thus from the point- of- view of a study of word co-occurrence 
and association, as the corpus size is increased from the 1500 abstract level 
its value, in terms of the number of high co-occurrence frequencies, increases 
far more than proportionatelyo 

11103 The ACE computer 

All the work involved in analysing the abstracts and computang statistical 
measures of word pair association was executed on the ACE computer o This was 
the only machine of its kind, having been designed and built at NPLo Since 
the experimental procedure has been influenced in a number of ways by 
considerations of what was and what was not feasible using that machine, a 
brief description of it follows o 

ACE was a valve-operated machine having mercury delay lines as its fast 
access store So The total delay line capacity was about 800 words, including 
24 long delay lines each holding 32 wordSo The word length was 48 bitSo Each 
instruction occupied one machine word and contained four addresses, two 
specifying the location of operands, the third specifying where the result 
should be sent and the last pointing to the next instruction to be obeyedo 
The cycle time of the 32 word delay lines was one millisecond. The time 
taken to transfer a word from one of these to another store or to the 
instruction register was therefore between 32(is and 1 ms depending upon the 
position in the delay line of the word being transferredo 

The long transfer feature of ACE was particularly useful for the analysis 
of word-pair frequencies. It allowed the contents of any number of words of 
a long delay line to be regarded as a single number or bit string. Thus the 
addition of two such numbers or any logical function of one or two strings of 
up ''o 1 ,536 bits could be executed in a single instruction. The time required 
to obey such an instruction was 1ms when the contents of the full delay line 
were involved and correspondingly less on other occasions. This facility 
was used extensively for binary matrix processing, 

32,000 words of backing store were provided by four magnetic drums. The 
information on each drum was held on 256 tracks, each holding one 32-word 
block. The drams had moving read/write heads. The average time for a track 
selection involving a head movement was about 35 niso Transfer of information 
between a dr'am track and the fast store took a further 7ms o 

The number of stems in the dictionary was limited to 1 ,000 so that a 
word/ abstract matrix could be stored on the drums with one track assigned to 
each word or matrix row, the remaining 24 drum tracks being available to hold 
program and other small amounts of datao 

A lower level of storage was provided by six magnetic tape decks. All 
full-valued (i.eo non-binary) matrices of word pair frequencies, associa'’ion 
factors, similarity coefficients, etc,, were stored on magnetic tape, their 
requirements vastly exceeding the capabilities of the drums. Constructing 
and processing such matrices was a relatively slow business because of the 
slow speed of tape operationso 

1 11 04 Matrix of word co-occurrence frequencies 



Constru.ction of this matrix involves applying the logical or Boolean 
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operation, AND, to two strings of bitso This is illustrated in Pigo IIIo4o1 , 

A and B being two given strings of bitso C is the string of bits produced 
3'PPlyi^^S the AliD operation to A and B, and contains a M * in every position 
in which both A and B contain *1 *o 

The first step in obtaining the co-occurrence frequency of two words is 
to produce the string of bits obtained by applying the AKD operation to the 
corresponding rows (stored as strings of bits, each row on a separate drum 
track) of the word/abstract incidence matrixo The co-occurrence frequency 
of the words is then obtained simply by counting the number of 1 *s in this 
new string of bitSo 

As mentioned earlier, the word/abstract matrix is so large that it has 
to be handled in sections, only one of which can be stored on the drums at a 
timeo The matrix is composed of eight sections, each having 1,536 columns 
and representing that number of abstractso 

A matrix of word co-occurrence frequencies is produced from each section 
of the word/abstract matr:J-X and is stored on magnetic tapeo Finally, the 
matrix of word co-occurrence frequencies for the full corpus is produced as 
the sum of these eight matriceso This was done by repeated application of a 
program written to sum two such matrices. When adding two matrices corres- 
ponding pairs of elements are added to obtain each element of the sumo 

Bit string A: 01 0001 1 01 01 001 1 1 001 0 

Bit string B: 1 001 1 1 0001 1 01 01 01 1 1 1 

Bit s bring A AND B: 000001 00001 0001 0001 0 

Pigo IIIo4o1 Performing the AND operation upon two bit strings 



IIIo5 Degree of association and statistical confidence 
We are interested in two things: 

1o statistical measurement of the degree of association between 
two words, 

2o the statistical confidence with which a degree of association 
can be asserted. 



In the general case these are independent. For an imderstanding of this 
consider the following analogous situationo It is generally accepted on 
statistical grounds that an association exists between cigarette smoking and 
the ducidence of lung cancer. The degree of association is veiy low, since 
only a very small proportion of smokers e.re affected in this way and by no 
means all lung cancer is attributable to this cause. However, because the 
statistics have been gathered from a vast number of case histories the 
evidence is overwhelming. The possibility that chance events alone account 
for the observed statistics and that no association exists can, for practical 
purposes, be ruled out. In this case we say that the association is 
assertable with high statistical confidence. 




When trying to establish whether or not two variables are associated the 
usual procedure is to start with the null hypothesis that they are not and 
then to apply a test to see whether this hypothesis can be rejected at a 
suitably hi^ confidence level. The chi-square test is often used for this 
purpose (see, for example, [2l]). It is inadequate for our purposes, 
however, since it v/ould only enab'Le us to establish whether or not two words 
were associated in their use, saying nothing of the degree of association. 
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Ideally we would like to be in a position to say at what confidence level we 
could accept any given measured degree of associationo This calls foi* rather 
more elaborate methods which would be costly in terms of oompui;er time to 
impl ement o 



It would be particularly important to consider confidence levels if one 
were working with a small s€Lmple of texts and were thereby forced to Trork 
with very small word co-occurrence frequencieso Under such circumstances 
there would be considerable risk that measures of high association were not 
significanto However, we have rejected all word pairs occurring once only in 
our collection and, by setting a threshold level upon our association measure, 
we cream off only about 1 0^ of the word pairs occurring at least twice© From 
Figo IIIo2o1 it is evident that many of the word pairs creamed off will have 
occurred twenty or more times© ¥e feel that under these circumstances the 
statistical significance level of selected word pairs is not of such vital 
importance and that one is justified in' considering only the measured degree 
of association© This is what we have done© 



III ©6 Choice of association factor 



One classical measure of association (see, for example, [2i] ), or 
association factor, is obtained in this context as the ratio of the observed 
number of co-occurrences of two words to the expected number assuming 
statistical ^independence (i©e© no association)© This may be expressed as 

N©n. . 

A.F.1 = ^ 

n. ©n . 

where N is the total number of abstracts, n. . is the number of abstracts 
containing both the ith and jth words, and 

n^ and n. are the respective numbers of abstracts containing the ith word 
and the jth word© 

One interesting property of this function is its symmetry© In evaluating it 
for a given pair of words it is of no consequence which word is considered 
as the ith word and which one as the jth, the same value of association will 
result in either case© Although, at first sight, this may seem entirely 
logical, there are grounds for arguing in favour of a non-symmetric measure 
of association yielding, in general, two values for each pair of words© This 
seems sensible when one observes, for example, that given a request including 
the word photography, relevant documents are quite likely to contain the word 
film, whereas documents relevant to a request containing the word film are far 
less likely to contain the word photography because *film* is used in many 
other contexts© Thus, if the purpose of an association measure is the selection 
of documents containing words associated with those in a given request, 
symmetry is not necessarily a desirable property© 



The conditional probability, p(wV^*)> finding word w< in an abstract 

1 j 1 

known to contain the word w. provides a crude measure of association which 

reflects this asymmetry in the use of wordso In general the probabilities 

p(w./w.) emd p(w./w. ) assume different values© 

1 0 0 1 




The association factor is computed for all word pairs and the I'esults are 
arranged in the form of a matrix© Quite a lot of processing requiring ready 
access to both the rows and columns is done with this matrix stored on magnetic 
drums© This presents a problem since, if the rows of the matrix are stored on 
successive drum tracks, then reference to the coliomns is slow and vd.ce versa© 

For this reason we were obliged to adopt a symmetric measure of word association© 
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The ith column and ith row of the matrix are then identical and the need to 
refer to both rows and columns is removedo We wished, however, to choose as 
an association factor A function which would assume a high value if at least 
one of the two conditional probabilities for a pair of words was large o The 
simplest function we could imagine satisfying these requii*ements is that 
obtained by selecting the greater of the two conditional probabilities, 

i.e. MAX[p(w^/wp, p(w^/w^)] . 

Tais function is not entirely suitable because, in the case of a very 
frequently used word, the conditional probability of that word given another 
word is likely to be high for many other given wordSo In this case a high 
conditional probability does not necessarily indicate a strong association 
between two wordso Thus, each conditional probability p(wi/wO should be 
considered in relation to the unconditional probability, p(w^), of the same 

wordo This cannot be done by taking the ratio of the two probabilities since 
that yields the first association factor we considered (A-^), which does not 
have the desired property of assuming a high value if either of the conditional 
probabilties is largeo We therefore followed the other obvious course and 
took the difference between each conditional probability and the corresponding 
unconditional one. Hence the function becomes 



MAX[(p(w^/wj)-p(w^)), (p(wj/w^)-p(wp)] . 

In terms of 



N, 



n. 

X 

n . 

3 



3 

3 



the number of abstracts in the collection, 

” " " " containing word w. 

ff II II II II II _ 

w . 



and n. " " 

this can be expressed as: 

MAX 
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Tne fact that we were not interested in negative associations enabled us to 
simplify the function slightly. As mentioned earlier we wished to reject all 
word pairs occurring less than twice, and we decided to do this by assigning 
the value zero to the association factor when n. . is less than twoo The 

form in which the association factor was finally computed is: 



AoFo5 



n. . 



MAX(n^,n.) , 



^ \ MIN(n^,np 
I 0 when n. . < 2 



N 



when n. . > 2o 



It is worth pointing out that this measure takes no account of 
frequency of occurrence or of co-occurrence within an abstract. There are 
two reasons for ignoring theses 

(i) dealing with short abstracts the important thing is that a word 

is introduced at all, the frequency of use within an abstract not 
having much significance and being partly a matter of style, and 



O 
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(ii) if techniques of this sort are ever to be of practical use the 

computing time required for processing word pairs (of which there 
might be an enormous number) must be kept to a minimumo 



IIIo7 Thresholding the association factor 



Computed as above to an accuracy of 15 bits, the association factors 
for all word pairs are packed, three per machine word, and stored on 
magnetic tapeo Access to information on magnetic tape is very slow, and the 
necessary processing of the association factor matrix would not be practicable 
if attempted in this wayo Instead, a threshold value is set upon the 
association factor, two words now being regarded as associated if their 
measured association is at or above this thresholdo In this way a binary 
association matrix is produced containing a *1 ^ to indicate each word pair 
association and zeros elsewhere o This matrix, being far more compact, can be 
held on the magnetic drums and processed in very much less time. Bearing in 
mind the purpose of the associations, we thought that the original matrix 
contained enough redundancy for this thresholding process to be quite 
sensibleo 



The problem, of course, is to decide what threshold value to seto We 
finally adopted the somewhat arbitrary procedure of assioming, on the basis of 
the proposed use of the resulting binary matrix, what would be a reasonable 
average number of associations per word, and setting the threshold value 
necessary to produce approximately that numbero This is achieved by first 
producing a distribution of the association values in the unthresholded matrix, 
illustrated in !Pigo IIIo7o1 o Starting with the highest association value, 
cumulative totals are produced of the number of associations exceeding a 
progressively lower threshold value, until the desired totail is approximatedo 
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We initially thought that we could decide what threshold value to set by 
studying the distribution of association factors produced from a constructed 
set of random * abstracts^. Although the exercise did not solve the 
thresholding problem the insults obtained are very interesting and they are 
described in the next sectiono 

IIIo8 Comparison of real and random abstracts 

A set of 1 ,536 random ^abstracts* was constructed having the same 
distribution of word frequencies as a batch of real abstracts# In fact we 
simply produced a random word/abstract matrix having the same number of 1 ^ s 
in each row as the real matrix (and therefore the same word frequencies) o 
The positioning of the 1 *s within the rows was determined by use of a 
pseudo- random number generating program# 
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This fictitious word/abstract matrix was processed to obtain the word pair counts 
and the set of association factors for all word pairs. 

Distributions of word pair frequencies for the real and random abstracts 
appear in Figo III080I0 They are remarkably similar. However, the real 
abstracts contain significantly more high frequency (i.e. commonly occurring) 
word pairs than the random, as shown by the differences in the cumulative 
totals. The total numbers of word pairs occurring are almost identical in the 
two cases. Fig. IIIo8o2 shows the distributions of an association factor. 

The association factor used here is a modified foim of AoF.I (section 111.6), 
the logarithm having been introduced to compress the range of values of the 
function. Again there is a significant, though quite small, difference 
between the two distributions. 

Anticipating a far greater separation of the distributions of association 
factors, vje had thought that a suitable threshold value might be obtained by 
taking a value exceeded by, say, only of the word pairs in the random 
abstracts. The hope was that such a value, used as a threshold for the real 
data, would reject a similar proportion of the fortuitous associations. The 
similarity of the distributions shows this argument to be invalid, or at any 
rate not to yield a useful resud-t. 

III. 9 Computation of similarity coefficient 



The association factor has two weaknesses with respect to the present 
application. The first is that we can be fairly sure that many high associations 
will occur by chance, in spite of the precautions we are taking. Secondly, in 
the short abstracts we are analyzing it is unlikely that synonyms or near 
synonyms will tend to co-occur, an author choosing one or the other, but having 
little opportunity of using both. Synonyms are therefore unlikely to be 
detected by means of their association factor. 

The similarity coefficient was introduced in the hope that it would be 
better in these respects, and because we think it has far more intuitive 
appeal as a measure of likeness of words. 



To compute the similarity coefficient between two words we com-pare their 
rows in a hinary matrix of associations . In this way we are comparing the 
sets of words found to associate with each of the given words, and if they 
are similar we want the similarity coefficient to be high. This is exactly 
analogous to the process of computing an association factor, in which two rows of 
the word-abstract matrix (also binary) are compared. We have therefore 
employed the same function (AoF. 5) used as an association factor. The reason 
for now referring to it as a similarity coefficient is to avoid ambiguity. 



The steps involved are: 

1 o Produce binary association matrix from matrix of association factors, 
AoF.5, setting a threshold value of 5/64® 
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Compute similarity coefficients as 



SoCoI 



i 



m. 



MIN(mi,m .) , 



MAX(m.,m.) 



0 



when in. . < 4 



when m. . ^ 4o 



where M = number of words in vocabulary = 1000 

m. ,m. = number of words associated with word i,j respectively 

m^, ^ = number of words associated with both i and j. 

^ j 
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Figo IIIo8„1 Distribution of ■word pair frequencies for real and 
random abstracts o 
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Note that the similarity coefficient is defined as being zero for pairs of 
words having fewer than four common associateso The matrix of these coeffici- 
ents is stored on magnetic tape in the same way as the matrix of association 
f actorso 
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Illoi 0 Word connection matrices, G-3 and G4 

!Two binary word connection matrices were used for producing word clusters 
for certain retrieval processes, in others they were used directly o One of 
these, G-3> was produced, as a hybrid from both matrices AoFo5 and SoCd • A 
complete print out of G-3 appears in Appendix A5« The other, G4-, was derived 
from S.Co1 alone (and, therefore, is just another word similarity matrix) o 

We thought it a good idea to produce one matrix as a hybrid so that any 
useful word connections brought out in the association matrix that may not be 
reflected in the similarity matrix might be retainedo The general procedure 
used to form G-3 was to threshold the A.Fo5 and SoCd matrices and then to 
combine the resulting binary matrices to produce a new binary matrix containing 
a * where either one or both contain M** When thresholding, the simple 
process of setting a fixed threshold value for an entire matrix proved 
unsatisfactory© This way binary matrices were produced with an excessive number 
of 1 ^s in same rowso These occurred in the case of words used very frequently 
in the abstracts, such as CIRCUIT, FREQUENCY, &EOPHYSICS, IONOSPHERE. The 
method used consisted in setting a nominal threshold value and, whenever a 
row was encountered having an excessive number of values above this threshold, 
raising the threshold for that row sufficiently to limit the number of 1 ’s per 
row in the binany* matrix produced to a specified maximumo The successive 
steps in producing G-3 were: 

1o Produce a binary matrix from SoCd using a nominal threshold value of 
29/64> and limiting the number of 1 *s per rov/ to 1 Oo 

2o Produce a binary matrix from AoFo5 using a nominal threshold value of 
15/64> and limiting the number of 1 *s per row to 5© 

3o Combine these binary matrices using a logical OR opern lion on pairs 
of corresponding elements, producing another binary matrix, Xo 

4o Transpose Xo 

5. Combine X and the transpose using a logical AND operation on pairs 
of corresponding elements. 

6o Finally, produce G-3 from the matrix produced in step 5 by giving all 
diagonal elements the value *1 

Steps 4 and 5 are included to make G-3 symmetric, the thresholding process in 
steps i and 2 destroying the symmetry of the original matriceso This ti. resholding 
gives higher priority to matrix SoCd than to AoFo5> only the highest 
association factors being permitted to influence G-3o 

After using G-3 for some time we decided, on the basis of results produced, 
that it contained rather too many non-zero elements and that another matrix, 

G4, should be formed with fewer© We decided, partly on the basis of some 
evidence that most word pairs with high SoC©1 also have high AoFo5> to base 
this only on matrix S 0 C.I 0 We wanted to produce a matrix with a more uniform 
distribution of 1 *s in its rows than G-3© For this reason the thresholding 
procedure was modified again. The steps involved in producing G4 were: 
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1o Prjduce a binary matrix from SoCd by setting a threshold value for 
each row so as to obtain a number of 1*s in each I'^ow as close as 
possible to 7o 

2o Transpose the resulting binary matrixo 

3o Conibine the matrix and the transpose using a logical AND operation 
on pairs of coiTesponding elementso 

4o Finally, produce G4 from the matrix produced in step 3 ty giving 
ail diagonal elements the value *o 

Matrix G3 has 188 rows in which only the diagonal element is non-zero 
(ioeo there are 188 completely isolated words), a maximum number of non-zero 
elements per row of 15 anti a total of 3^582 non-zero elements. G4 has 112 
rows in which only the diagonal element is non-zero, a maximum of 7 non-zero 
elements per row and a total of 3^074 non- zero elements. 
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IV Generation of ViTord Clusters 

IVo1 Graph representation of word connections 

The methods we use for word clustering assume as a starting point a set 
of binary word connections, i.eo connections which simply do or do not exist 
between words, having no associated weights or strengths. We have thought of 
these connections as being represented by a square binary matrix having rows 
and columns corresponding to the words and element values 0 or 1 to indicate 
the absence or presence of connections. 

When considering problems of clustering we have found it useful to 
imagine this sort of data being represented by a graph o Fc"* this purpose we 
define a s^aph as a set of points and a set of lines connecting some or all 
pairs of pointso Consider a graph having a unique point representing each word 
and lines representing a given set of word connections. The problem of 
finding clusters of words amounts to locating subsets of points in the graph 
which are fairly densely interconnected relative to the graph as a whole, 

IVo2 Properties of clusters 

The sort of properties of clusters in wliich we are interested is their 
number, size and degree of overlapo Having no means of knowing what values 
of these parameters are desirable for retrieval purposes, we decided to 
produce cluster sets exhibiting different characteristics and to test their 
retrieval capacities experimentally o 

To some extent the properties of clusters derivable from a given set of 
word connection data are determined by the datao However, they are also 
influenced by the choice of definition of clustero For example, a particular 
definition might not permit common membership of clusters, thereby restraining 
all clusters to be mutually exclusiveo We have therefore tried three quite 
differrnt methods of producing clusters, with three different implicit cluster 
definitions, 

I Vo 3 Completeness of clusters 

Each word in a completely connected cluster is connected to all other 
words in the cluster. That part of a graph corresponding tc words and 
connections of a completely connected cluster forms a complete subgraph of the 
containing graph. If it is contained in no larger complete subgraph it is a 
maximally complete subgraph,* 

In the case of word clusters based upon our statistical associations 
completely connected clusters containing more than three or four words are 
unlikely to emerge from the raw statistical data. Unless the connections 
specified by the raw data are supplemented in some way by further processing 
prior to cluster detection it is therefore sensible to adopt a definition of 
cluster which does not demand complete comectivity, 

IV,if Fragmentation of a graph by random removal of lines 

The basic idea here is to break down the graph of connections into 
independent fragments by progressively removing lines from the graph, the 




* Sometimes called a clique . 
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selection of lines to be removed being made with the aid of an algorithm for 
gene ‘eating pseudo-random numberso A set of fragments is independent in this 
sense if the points and lines in each are such that no connection exists 
between points in different fragmentso An assumption underlying the method t. 
be described is that when a graph is broken down in this way connectivity is 
more likely to be preserved in those parts of the graph corresponding to 
clusters being sought (these parts of the graph having a relatively high 
density of connections) than elsewhereo 

The graph defined by connection matrix G-3 (section III0IO) was examined 
to find its independent fragments o Apart from isolated words, there were two, 
one containing only four words, the other containing the rest© Independent 
fragments containing ten words or fewer were removed from the graph and K^o of 
the remaining connections were selected remind omly and eliminated. The 
resulting graph was examined and again independent fragments containing ten 
words or fewer were removed. The processes of eliminating a reindom selection 
of connections and then removing from the graph any small independent fragments 
were repeated until tlie entire graph had been reduced to a number of such 
small fragments, fragments obtained by this process containing from four to 
ten words each were storedo The complete procedure was repeated 21 times, 
each time starting with the graph of G-3 and eliminating different random 
selections of connections, until 1,000 fragments were accumulated in the size 
range four to ten words. 

At this stage we were likely to have obtained a number of fragments 
approximating to each cluster of words in the original graph. The processing 
that followed was intended to reconstitute the clusters by first comparing the 
1 ,000 fragments accumulated, identifying sets of similar or overlapping 
fragments and merging them. 

A few words featured rather insignificantly, each appearing in only one 
or two of the 1,000 fragments. These words were discarded. A similarity 
matrix was produced showing similarities between pairs of fragments. The 
measure of similarity used was the number of words that two fragments had 
in common divided by the number of words in the smaller. Different thresholds 
were set to produce binaiy similarity matrices which were processed to find 
independent sets of fragments, i.e. sets of similar fragments having no 
similarity connections between members of different sets. A similarity 
threshold of 19/32 was selected which maximized the number of independent 
sets of fragments, producing I4B0 Each set of fragments was merged to form a 
cluster by including in the cluster any word appearing in at least half of the 
fragments in the set. The word clusters thus produced were not mutually 
exclusive since the sets of fragments from which they were produced, though 
independent with a similarity threshold of 19/32, would not all have been 
independent had a lower threshold value been used. 

446 distinct words appear in the 148 clusters formed. These clusters 
together with the 554 other words (used singly) constituted cluster set C4 used 
in retrieval run 19. 

IV. 5 Meetham’s method of clustering, based upon removal of lines not 
included in increasing numbers of triangles 

Here a basic assumption is that, for indexing and retrieval- purposes, 
word groups should be mutually exclusive and that words which are associated 
with two otherwise weakly related or non-related word groups constitute an 
undesirable ambiguity. 

Word connection matrix G-3 was the starting point for this clustering 
experiment and, with the exception of I88 completely isolated words, it 
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consists largely of a set of words from each of which paths of connections can 
be traced to eveiy other word in the set. The aim was to break down the graph 
of word connections into mutually exclusive components, the words in which 
satisfy intuitive notions of word clusters as regards their size and semantic 
range o The following procedure, used to obtain this brealc-down, is described 
in [i o] , while [S] and [ll] are also of interest in this connection. 

All lines (connections) were removed from Gr3 which were not included in 
at least one triangle (of connections). This split the s^aph into exclusive 
components containing 413^ 298, 134^ 81 ^ 18 , 12, 9^ I 9 6, 5^ 5^ 3 and 3 lines 
respectively. Lines were removed from the four largest components if they 
were not included in at least two triangles. This yielded 15 components 
containing 3 to 1 8 lines and 7 larger ones, the largest having 167 lines. 

The process of removing D,ines unless they were included in at least two 
triangles was applied iteratively to the 7 larger components until no 
further changes occurred. (An interesting theorem about the convergence of 
this iterative process is considered in [ll3)» This yielded 14 components 
having 6 to 1 8 lines and 14 having 20 to 75 lines. 

So far G-3 had been split into 53 small but strongly oonneoted oomponentSo 
It was still desirable somehow to partition the 14 larger ones. Inspection 
showed that the removal of one or a small number of points from each 
conveniently partitioned them further: the points removed corresponded to 

the ambiguous words mentioned earliero It was then necessary to decide 
whether to leav ? those removed as further isolated points, whether to link them 
up in some way or whether to include them in any of the word groups® 

The resulting set of v.'ord clusters (set C2 used in retrieval run 16 ) 
included 491 isolated words and 122 clusters each containing between 2 and 
1 5 werdso 

IV. 6 Cluster emphasis * 

It is assumed that, in the graph of word coxmections, the subgraphs 
representing clusters being sought are unlikely always to be complete® An 
iterative procedure is used to process the original graph in such a way that 
clusters should be emphasised, connectivity within them bein^ increased and 
that between them reduced, to a point at which it is possible to detect them 
as maximally complete subgraphs® 

Consider a binary connection matrix (section III 0 IO), C, with 1,000 rows 
and columns corresponding to the word stems being clustered, and element values 
of 1 and C representing the presence and absence of connections respectivelyo 
Consider also the corresponding graph, &o In most clusters there will be 
pairs of words which are not directly connected. Some such pairs are likely 
to be connected by paths of length two (ioeo via a third word). The 
hypothesis underlying the technique described is that, considering all pairs 



*An account of this part of the work was given at the 1968 IFIP Congress at 
Edinburgh and is published in the Proceedings [ 9 ] 0 

’^We used Wolfberg^s algorithm for finding all the maximally complete 
subgraphs of a given grajjh. A reference to this, [22], appears in the 
bibliography. 
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of words not directly connected in G-, the proportion of such pairs 
connected by paths of length two is likely to be higher for the set of pairs 
occui*ring within clusters than for other pairs o 

The first step of the emphasis procedure is to form a new non-binary 
connection matrix the same size as Co A direct connection in G- between a 
given pair of words contributes p towards the value of the corresponding 
element in the new matrix, and each path of length two between the words 
coatributes Xo The new matrix is computed from C as (pC+XC^), integer 
values of p and X being specified when the program is runo A binary form, 

C^, of the new matrix is produced by thresholding its value So The relative 
contributions of direct paths and paths of length two in G- to the 
connections in the graph G* , corresponding to C*, depend upon the values 
chosen for p and Xo Suppose that the threshold value is chosen to make 
the resulting number of connections in G' as close as possible to the 
number in Go If the above hypothesis is correct the density of connections 
within clusters will be higher in G’ than in G and, of necessity, the 
density of connections elsewhere will be lower in G' than in G. The 
desired emphasis of clusters will thus be achievedo 

Using G3 and Gl\. (section III 0 IO) as initial connection matrices, this 
process was applied iteratively with various values of p and X and trying 
different methods of thresholding, including that outlinedo 

Tae process seemed fairly convergent in the cases tried. Two sets of 
maximally complete subgraphs (mCS's) were used as descriptorso One set, 
produced from G3, contains 1,178 highly overlapping clusters. This 
includes 1 88 isolated wordSo These were produced by a single application of 
the emphasis process with p = 2, X = 1 and a threshold value of 2o This 
is the cluster set, C3, referred to in section VIo2.2 and used in retrieval 
run 14o A sample showing the very high degree of overlap of clusters in this 
set appears in Appendix A 60 The set was selected because of this characteristico 
The size disoribution o! tnese MCS’s and of those of G3 appears in figo IV. 60 I 0 
The size of the largest MGS is increased from 6 to 1 2 and the number of MCS*s 
of 5 or more words is increased drastically by the emphasis process. However, 
the n\imber of connections in the graph has increased approximately five-fold o 
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?igo IV 060 I Size distribution of fiCS*s before and after 
emphasiso 

The second set, cluster set C5 used in retrieval run 20, was produced 
after five iterations of the process, starting with connection matrix G4, with 
p = X = 1 and thresholding to maintain a number of 1 in the matrices as 
close as possible to the 3^074 in G4o Figo IV.6o2 shows the distribution of 
sizes of the MCS*s of the original graph and of those obtained after the 
iterationso 
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Pigo IVo 6*2 Distribution of maximally complete subgraphs produced by five 
iterationso 

The convergence -evident from figo IVo6o2 is typicalo Two typical 
clusters obtained in the final iteration are shown in figo IVo6o3^ the connec- 
tions being those in &4- before any emphasiso 





Pigo IVo6o3 Two clusters obtained from &4- by five iterations of the emphasis 
processo 
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PART 3 • Retrieval libcperiments 
Vo G-eneral Consideration s 
V.1 Brief description of KDF9 

KDP9 is a medium size, general purpose machine having a 48-bit word 
length o The main fast store, or core store, can be shared by up to four 
programs running at fixed i^rioriiy levels determined by the operatoro The 
core store has fetch and store times of six microseconds o The times taken 
for the fixed point operations of addition and multiplication, exclusive 
of fetch and store, are 1 and 14 microseconds respectively q 

Special features of the machine are its two 1 6-word nesting stores, or 
stacks; one used to store the operands and results of any functions 
performed, the other used to hold the link instructions for nested subroutines o 

The NPL machine has the following configurations 

32,000 word core store, 

6 magnetic tape units, 

4,000,000 word disc backing store, 

Line printer, 120 character line, 600 lines/mino, 

Paper tape reader, 1,000 characters/sec.. 

Paper tape punch, 110 characters/seco , 

Card loader, 600 cards/mino, 

Typewriter control terminal o 

Vo 2 Search requests 

When published, the abstracts used in these experiments appeared under 
five broad subject headingSo The distribution was unbalanced even under 
these headingSo We therefore had no knowledge of the subject profile of the 
collection. In collecting a set of search requests we naturally wished to 
achieve a reasonable correspondence between the distribution of their topics 
and that of the subjects in our document collectiono We were thus faced 
with the problem of trying- to match this unknown subject profile o We decided 
to do this by basing the requests upon a set of abstracts arbitrarily 
selected from the collectiono 

It was thou^t that a set of 1 00 requests would provide a reasonable 
spread of subjects o This number was obtained from 21 people, the number 
per person varying from two to seveno Each person was sent a set of source 
abstracts each of which he was told to consider simply as specifying the 
subject area of one search requcsto He was told that it was immaterial 
whether or not a source abstract was a relevant answer to the request based 
upon it o 

The requestors were also informed that the systems under test were based 
entirely upon key -words and that all numerical data and mathematical symbols 
had been omitted when key -punching the abstracts o In accordance with this 
they were asked to exclude from their requests numerical data, symbols other 
than those in the Roman alphabet, and abbrevlations o 

A complete listing of the requests appears in Appendix B 1 o It is 
evident that the exclusion of numerical data has in some requests led to the 
use of words such as high and low where, for example, a frequency range or 
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attenuation factor might otherwise have been specified more precisely. 

However, this sort of thing is noticeable in relatively few cases. 

Some time after eliciting the requests we wished to tiy out a 
retrieval strategy (see section VL2,6)in which a specified word or words 
in a request were regarded as obligatory in that only abstracts containing 
such words would be retrieved. We asked the requestors to consider their 
requests and to indicate by underlining any word in them for which they 
were fairly confident that there was no suitable substitute. 

Only 93 of the original set of 1 00 requests were used. However, the 
other seven are included in Appendix B1, together with the reasons for 
their cancellation, in Appendix 53° 

Vo 3 Form of output 

For each request, the output from each retrieval run includes the full 
title and abstract of all items retrieved in answer to the request that have 
not been so retrieved in any previous run . Thus, the first time a particular 
abstract is retrieved in response to a given request it gets printed and sent 
to the author of the request for relevance assessment. If the same abstract 
is retrieved in a later run in response to the same request it is not 
printed outo This minimizes the burden placed upon the requestors to assess 
relevance o It means, however, that the relevance assessments must be fed back 
to the computer for reference in later runs (see section Vo 7 on the harvest 
file). 



A sample showing the output format appears in Figure Vo3o1o The request 
is printed, followed by the retrieved abstracts. The first line of each 
abstract is actually the document titleo The serial number of each abstract 
appears on the left, and to the right of each is repeated request niunber/ 
run number/serial number, with a space for the requestor to insert a number 
indicating his relevance judgement. The output continues on as many pages 
as are required for the retrieved abstracts. In runs utilizing the 
underlining of request words the underlining is indicated in the print out 
by a string of X’s beneath the word or words in question. 

The ultimate page of the output for each request is a s ummary sheet 
which simply lists the serial numbers of all the abstracts printed out for the 
request with spaces for the insertion of relevance assessments. These 

summary sheets were not sent with .the rest of the output to the requestors. 

When the relevance assessments for a run were available the information was 
copied on to the simomaiy sheet So It was found useful to have the information 
in this compact form when preparing paper tapes to feed it back to the 
computer for storage in the harvest file (appendix B6)o 

Vo4 Batch processing 

The indexing and retrieval operations involve processing files, many of 
which are held on magnetic tape and some of which are rather large o The 
abstract/word matrix is an example of a large file, consisting of 11,571 
lists of word numbers, one corresponding to each abstract indicating the words 
it contains. These are written on to magnetic tape in abstract serial 

number order. They occupy about 20^ of a reel of tape and take nearly one 

minute to read or write. When such a file is being processed the time 
occupied by magnetic tape movements generally far exceeds the actual computing 
time incurred. This means that for much of the processing involved in 
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indexing and retrieval only a marginal increase in total computer time is 
required to process a batch of 100 requests instead of a single oneo Consider, 
for example, the. process involved in comparing the sets of key -words occurring 
in the abstracts with those occurring in the requests (the main part of the 
basic key -word strategy described in section Viol) o The 100 key-word sets in 
the requests occupy little space and can easily be held together in the core 
storeo When this is done it requires only a single scan of the much larger 
file indicating the set of words in each abstract in order to compare all 
requests with all abstractso The same amount of tape-scanning would be 
required to process a single request© 

Another advantage cf batch processing the requests is seen in the final 
stage of the retrieval process where it is necessary to assemble the particular 
selection of abstract texts to be printed out for each request© The complete 
file of texts fills a 2,300 ft reel of magnetic tape© It takes about four 
minutes to scan this file and hence that is approximately the time that would 
be required to assemble the abstracts to be output for a single request© 

With batch processing we begin by reading the entire file of abstract texts 
from tape and dumping them on to the magnetic disc© This operation takes 
about 12 minutes and would therefore be pointless when processing a single 
request© Once on the disc, the abstracts can be accessed randomly in very 
little time© The requests are then considered in sequence, the abstracts 
required for each are read into the machine, arranged in the required format 
and printed out© 

V©5 Possible Variables 



Various methods have been suggested for improving key-word stem per- 
formance independent of word-associations, not all of which we have wished 
to try© 

The principal are:- 

(1 ) Weighting individual request words, by reference to the requestero 

(2) Weighting individual document words by counting their repetitions. 

( 3 ) Using functions such as the 'cosine’ to measure the degree of 
match. 

( 4 ) 'Underlining', that is, insisting that one or more particular 
request words be present in each document retrieved. 

To examine (1 ) thoroughly by varying the weights would involve much 
more work for the requesters than we were prepared to ask them to do. 

However, a small experiment was tried in getting the machine to assign weights 
namely our Run 22 (AWKWS). ’ 

( 2 ) seems of doubtful value with short abstracts where a chance 
repetition is more likely to introduce noise. 

( 3 ) takes account of the length of the document, so that e.g. a document 
only partly on the subject of the request scores lower than one wholly 
devoted to it. 




^These^ methods introduce new variables which we have not been able to 
try in addition to our own work on association techniques. 

(4)_ though not appropriate to every request, we have explored, and we 
discuss it in section VI. 2.6, 
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Vo 6 Quanti-far of output aad cut-off 



Consider these totals from Appendix B7:- 

Run 1 3 o Coordination = Number of Key-word stems in Commono 
Request O p Coordination Number of Documents Output (Cujmulative) 



(6 keywords) 


5 


14 


14 




4 


87 


101 




3 


463 


564 




2 


1388 


1952 




1 


3639 


5591 




0 


5980 


11571 


Request 2. 


4 


2 


2 


(5 keywords) 


3 


26 


28 


2 


203 


231 




1 


1803 


2034 




0 


9537 


11571 


Request 3# 


3 


14 


14 




2 


228 


242 


(6 keywords) 


1 


2426 


2668 




0 


8903 


11571 






Pigo V. 60I 





That is, 14 documents have 5 out of the 6 keywords of Request 0, 87 have 4j 
101 have 5 or 4 and so ono (it happens that none have all 6)0 Since 
individual documents at any one coordination level are not distinguished, 
th^ must be output togethero Retrieval consists in 'creaming off* 
successive 'strata* from the library, which has been arranged as it were in 
a 'pyramid 'o 

We shall use 



stratum = non-empty set of documents at the same 
coordination 



output 

(when used as a noun) 



total (number of) documents, relevant 
or irrelevant, from the top coordination 
down to a given point inclusive, for one 
or more requests, and consisting of a number 
of complete strata, or zeroo 



As we have described in section Io14^ the customer specifies a cut-off 
value , which we denote by Ko He asks for as near as possible K documents , 
with, let us say, fewer rather than more in the case of a tie. Now 0 and 
2K are equidistant from Ko If the top stratum has 2K or more documents, the 
strategy is not sufficiently sensitive for the customer's purpose and gives 
no output o Thus in any case the output lies in the range 0 to (2K-1 ) inclusiveo 

Consider Figo V060I again, and set K = 20o 
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The outputs for requests 0,2 ,3 9> <>..> 99 are 14, 28, 14, 9, 7, 12, 

4j 0 . 0 , 33 whose average is only 14 >20, considerably less than 20o 

Similarly, if we ask for 50 documents, we get an average of 35 per 
request® 

We define 

K' = effective cut-off = average number of documents per request 
output for a particular run and a particular request set . 

In short, we ask for K documents per request and get K* on average « 

By inspection we find that (1 ) for any given run and our 93 requests, 
is approximately constant, 

K® 

Apprpximate value of 



Strategy 





13 


14 


K = 20 


o7l0 


.919 


30 


.739 


.931 


40 


.725 


o918 


50 


0697 


o 926 


1 00 


0650 


.959 


250 


0827 


.971 


500 


.732 


.956 


PiSo 


V»6„2 





and that (2) this constant is not necessarily unii^o 

Take an imaginaiy single request, with possible outputs of (say) 0, 3, 8, 
15, o o • , docioments c 

K 01 22 4-56 7891011 1213141^000 
K* 0031 3 38888 8 8l51515Hooo 




Pigo Vo6o3 

K’ 

1 ^- oscillates about the value 1 o If we have several requests, we might 

expect these oscillations to cancel to some extent, since the points at which 
the average K’ changes become more frequento This helps to explain (l ) but 
( 2 ) is surprisingo 

In Appendix B14 we show that ( 2 ) is a second- order effect which remains 
appreciable under the following sets of conditions, which, in our context, 
are roughly equivalent to each other; - 

(i) For any given request the ratio between successive cumulative 
output totals is quite large, say 3^1 or more® 

(iij The outputs nearest to K for the different requests have a 
large spread® 
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(iii) The strategy employs few coordination level Sr> 
In fact, if we can assume (l ) we show there that 



where v 



K'' 

K 

standard deviation 
me an ” 



1+v*^ 



for the outputs nearest to K, so that v 

K> 

when — iso 
K 



also is approximately constant 



Vo 7 T he sensitivity of a strategy to changes in the cut-off K 

A number of ideas put forward in the last section relate particularly 
to systems such as ours where the documents fall into strata of several 
or many at a timeo We say that a strategy is the more sensitive the more 
strata it has within a given output o For example, Run 13 (KWS) at K = 50, 

K* 35 has only 1o9 on average, and 2o0 at K = 71 j K’ 50o Run 
(MCS01 ) , a descriptor run, has 6o6 at K = 55j K’ 50o Thus a given increase 
in K is more likely to yield new output for Run 1A than Run 13r This is 
important in an interactive use where documents are produced, let us say 
on a screen, stratum by stratumo 

K® 

From Vo6 it appears that the value of if sufficiently constant, 

provides a useful guide to sensitivity, being nearer to unity the more 
sensitive the stra.tegyo llosely related are the ratios of successive 
output totals for single requests, and the spread of the outputs required 
to approximate to K over a set of requestso 

The effect of underlining (V1.2o6)is to remove part or all from each 
stratumo In general, the number of strata within a given output appears to 
decrease and the strategy becomes less sensitiveo This is more marked when 
Run 13 is underlined (Run 25) than when Run 14* is (Run 21 ) , as may be deduced 
from the tables of K and K* in Appendix B5o 

Vo 8 The harvest file 



This name was given to the file kept on magnetic tape and designed to 
store the results of every strategy as found, in as useful and flexible form 
as possibleo It contained for each request complete lists of abstract 
numbers retrieved within the cut-off by each successive strategy, their 
coordination and relevance, and a flag to mask the first occurrence of an 
abstract for a particular request© 

Standard programs recopied assessments known from previous strategies, 
and arranged that no abstract was sent for assessment twice in regard to a 
particular requesto This kept tne work of the assessors to a minimum, 
decreasing markedly as more strategies were run© It also allowed us to 
try out strategies related to combinations of earlier runs, even if the new 
abstracts were not sent for assessment, since we could calculate to what 
cut-off they were already fully assessed© 



O 
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The harvest file was basic to all our evaluation pro grains o These 
included studying selected requests, smaller cut-offs, and the sets of 
abstracts retrieved by one strategy and not by anothero A separate program 
calculated the minimum cut-off needed to retrieve each abstract by the 
particular sti’ategy; we would recommend storing this also on the harvest 
file in any future experiment, as each new strategy is included o Fuller 
details are shown in Appendix B6 o 
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Indexing and Retrieval Stratef:ie8 

VI o Description of strategies 

A sununaiy of this section will be found in Appendix B4 o 

The main kinds of strategy used for indexing and retrieval are 
indicated below: 



Strategies 




Figo VI 0 I 0 I Indexing and retrieval strategies employed 

A complete description of the strategies followso Most of the 
indexing and retrieval processes were implemented in terms of fairly basic 
matrix operations upon data array So Details of this matrix representation 
of the problem are not considered here, but are dealt with separately in 
Appendix jti6, together with the computer programs usedo 

Determination of cut^-off level 



o 
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Our original idea was to aim at the same cut-off level, K, for each 
request in each runo Thus, in the first I 3 runs (ioeo runs 0 to 12) the 
cut-off coordination value was always set so as to produce an output for 
each request as close as possible to 50 documents (ioeo K = 50)o 

As discussed in Section VIlc^ we later decided that K®, the average 
output per request, formed a better basis of comparison of different runSo 
Thus, in later runs, the average output was maintained as close as possible 
to 50 abstracts per requesto Nine of the early runs were repeated in this 
wayo In these cases only the repeat runs are quoted when presenting resultso 
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The cut-off procedure thus involved carrying out the processing for a 
strategy to the point at which coordination values had been established for 
all documents in relation to each of the 93 requests being batch processed 
and fonning tables similar to those in figo V 060 I o Programs were then run 
which systematically tried different values of K, the number of documents 
aimed at per request, starting at one and increasing to find the value that 
produced an average output per request, K®, closest to the desired value 
of 5 O 0 Having thus established the cut-off value to be set the necessary 
processing was continued to complete the particular strategy in hando 

The term standard runs is used to refei* to the set of runs in which K* 
has been made as close as possible to 50, and the output from which has been 
fully assessed for relevance. The standard runs are 6, 9j 11 and 13 to 23 
inclusiveo 

Basic key-word stem (KWS) strategy - run 13 

The key-word stems in a request are compared with those occurring in 
each abstracto The number in common with a given abstract is noted and taken 
as its cpordination value o The value of K is found giving a value 

closest to 5 O 0 The coordination value corresponding to this cut-off is 
determined and applied as a threshold to cream off the output for this 
re quest o All new material (ioe. abstracts in this output that have not 
previously been output for this request) is printed out for distribution to 
the requestoro 

Simple use of word clusters as descriptors (no representation) - runs M _ 1 6 , 
1 9 and 20 



The strategy here employs a set of word clusters, each cluster being 
used as a descriptor. The only difference between these runs is in the choice 
of word clusters, cluster sets C3, C2, C4 and C5 (see Section IV )being used 
in rims 14 ^ I 6 , 19 and 20 respectivelyo 

An abstract is indexed by assigning to it all clusters, from the 
particular set in use, with which it shares one or more key -word stems© 

Clusters are assigned in the same way to the requests according to their 
common key -word stems. From this point on the procedure is exactly 
analogous to the basic KWS strategy. The only difference is that the 
coordination value of an abstract in relation to a request is now the number 
of assigned clusters they have in commono 

Use of clusters as descriptors with un-weighted representation - runs 6 and 1? 

Most of the sets of word clusters employed as descriptors are such that, 
althou^ some words may appear in only one cluster, others occur in several© 
Thus, using the simple sort of strategy just described it is possible for two 
abstracts retrieved at the same descriptor coordination level to represent, 
by inclusion of the actual words or assignment of clusters containing them, 
different numbers of request words© We define the representation level of an 
abstract in relation to a given request as the number of request key-word stems 
actually occurring in the abstract or appearing in at least one cluster 
assigned to index it. 
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RADIO NOISE FROM PLANETS - F, HOrner, (Nature, London, Vol, 180, p, 1253; Dec, 7, 
1957) • A comparison is made of H,F, radio noise from Jupiter and Venus and from 
terrestrial liE htninR , The hypothesis that radio noise from the two planets is due 
to electrical di schar^es analogous tc terrestrial lightning requires modification, 

ABSTRACT A 



AN INVESTIGATION OF WHISTLING ATMOSPHERICS - L,RoO, Storey* (Phil, Trans, A., vol, 2^6, 
pp, 113-141; July 9, 1953) • A comprehensive report of an experimental and theoretical 
studj' of whistling atmospherics, at frequencies -^*15 kc is given, Whistlers may or 
may not be preceded by ordinary atmospherics, produced by lightning strokes at a 
distance of '*‘-'2000 km. The diurnal and annual variations of the properties of both 
types were investigated. Explanatory theory of their origin advanced by Eckersley is 
developed. Measurements of the degree of dispersion indicate an electron density in 
the upper atmosphere considerably larger than expected. This result is explained on 
the assumption that electrons are falling in from outside, and this might account 
for the relation between the occurrence of whistlers and magnetic activity, 

ABSTRACT B 
Fig, VI. 1,2 



An example is provided by request no. 36 for abstracts on s~i multaneous 
observations of whistlers and lightning discharges . Considering cluster set 
C3 the key -words, which are underlined, occur as follows 



SIMULTANEOUS in clusters 
OBSERVATION " " 



WHISTLER 

LIG-HTNING- 

DISCHARGE 



tt ■ tt 

tt tt 

tt tt 



712 and 713 

465, 466, 467, 468, 469 
595, 596, 597, 598, 599 
and 600 
326 and 1 032 
-110, 279, 346 and 547 
279, 28C, 23l and 282 



Thus, the request is formulated by 22 distinct descriptor clusters. 

In relation to this request it is interesting to note that (referring 
to Pigo VEo1o2) : 



Abstract A - contains 2 key -words of the request, has descriptor 
coordination level 14, has representation level 3o 

Abstract B - contains 2 key-words of the request, has descriptor 
coordination level 7, has representation level 4o 

Thus, although Abstract A is indexed by twice as many request descriptors 
as Abstract B, it represents fewer of the key-words in the requesto 



This strategy attributes importance to the representation level. The 
complete set of abstracts is ordered, placing^ abstracts representing the 
largest number of request words at the top of the list, followed by those 
representing the next largest number, etc. Abstracts with the same 
representation level are ordered, as in the previous strategy, according to 
their descriptor coordination level© The process of creaming off as nearly 
as possible to the required number of .abstracts is carried out as in the KWS 
strategy. 
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Use of clusters as descriptors with weighted representation - run 11 

In the previous strategy an abstract, A, represents a request word, W, 
if it is indexed by at least one word cluster (drawn from the set being used 
as descriptors) containing W, It seems reasonable to regard W as being 
more strongly, or better, represented by abstracts indexed by several clusters 
containing W than by only one. Accordingly, the criterion for 
representation is now modified sli^tlyo An abstract represents a request 
word if it is indexed by at least one third of the clusters containing the 
word. Apart from this the strategy remains unchanged o 

Word-stem comparison with expai ed abstracts - runs 15 aJid 18 

One possible advantage to be gained by loying word clusters to reflect 
word associations is economy of storage space in the computer. Rather than 
s.*ore all the individual word -pair associations one simply keeps a record of 
the word content of each cluster. In the case of a cluster containing n 
words. this means storing n items as against some number almost certainly in 
excess of - this and, possibly, as great as n(n-1 ) if the individual 
associations are stored. There is also the hope, which has a great deal of 
intuitive appeal, tliat the clusters one finds will represent in some way salient 
themes or concepts relating to the subject matter of one’s document collection. 

However, these possible advantages could well be offset by the undeniable 
fact that sensible clustering procedures, even crude ones, are rather costly 
in terms of computer time. 

We therefore thought it worthwhile testing procedures which do not involve 
clusters at all, but make direct reference to a word connection matrix. Two 
such matrices were used, matrix G-3 in run 15 and matrix G-4 in run 18. 

Each abstract is processed by noting -its key-word stems, referring to 
the connection matrix to find all the words connected to each of them, and 
adding these to the original words to form an expanded abstract. Key-words 
in' the requests are then compared with those in the expanded abstracts, 
continuing as with the basic KWS strategy. 

Word-stem comparison with expanded abstracts and requests - run 2^ 

Using connection matrix G4 the same method of expansion is applied to 
both the abstracts and the requests. The procedure is then identical with 
the basic KWS strategy^ 

Combined strategy ~ run 9 

This is intended to combine the advantages of the basic KWS strategy 
(run 13) and the strategy of run 14 which employs the set of word clusters, 

C3, and performs relatively well. These two runs are effectively repeated 
using a K’ value of 500 instead of the usual 50. For each request a new 
list of abstract numbers is produced containing those appearing in both cf 
the sets of approximately 500 thus produced. A new "coordination value" is 
assigned to each abstract listed obtained as the product of the values in 
runs 13 and l4o Finally a K* value of 50 is taken to determine the cut-off 
for each request resulting in an average output of 50 abstracts per request. 

Combined strategy - run 28 

Q This is a combination of the basic "MS strategy (run 13) and that used 
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in run l4o Coordination values are obtained as follows: 

Run 28 coord^o value = 2^ x (run 13 coord^o value) 

+ (run coord o value)® 

The effect is to retain the major division of the document collection 
produced in run l3o However, the document sets at each coordination level in 
run 13 are now ordered according to the coordination levels they obtained in 
run 14o The factor 2^ in the first term is greater than 99> the highest 
coordination value in run 1 thus avoiding any cariy between the terms in the 
above expression® 



Obligatory request words >- runs 21 , 2 l\, and 25 

Information regarding the word(s) in his request which, given the 
option, a requestor would make obligatory was used to remove from consideration 
all abstracts not containing the word(s)o The strategies of runs 10, 11 and 
13 were then repeated, in runs 21, and 25 respectively, but instead of 
retrieving from the complete set of abstracts the appropriate reduced set was 
used for each request® 



Automatic Weighting of Request Key-Word Stems - run 22 

A set of programs was written to display the request words which had 
been involved in the retrieval of particull.ar abstracts, whether directly 
(eogo Run 0, ioeo the basic KWS strategy with K’ = 35) or via some 
associated word (e.g. Run 11 ). It appeared, especially in the case of Run 0, 
that the request words often fell into two groups® 



Consider, for example. Request 53o 



ABSTRACT 


nURORMAT TRANSIST 


AMPLIP 


DESI&N 


DRIFT 


LOW 


176 


2 


314 


491 


520 




982 


973 


2 


314 


491 


520 




982 


1504 


3 


314 


491 




692 


982 


16A4 


3 


314 


491 


520 


692 




1646 


2 


314 


491 


520 




982 


3430 


2 


314 


491 


520 




982 


3567 


2 


314 


491 


520 




982 


4498 


2 


314 


491 


520 




982 


5370 


2 


314 


491 


520 




982 


6202 


2 


314 


491 


520 




982 


7019 


3 




491 


520 


692 


982 


8030 


3 


314 


491 


520 




982 


8590 


3 


314 


491 


520 




982 


8694 


2 


314 


491 


520 




"82 


8695 


2 


314 


491 


520 




982 


1 0505 


3 


314 


491 




692 


982 


10645 


2 


314 


491 


520 


692 




10776 


2 


136 314 


491 


520 






11467 


3 


314 


491 


520 


692 




11468 


3 


314 


491 




692 


982 


indentation marks 


the abstracts later 


assessed 


as relevant (3 = 


relevant 
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We notice that, if an abstract has 4 out of the 6 possible key-words, it 
is highly likely that 3 of them are Design Transistor and -Amplifier, and 
also that the fourth is LoWo The first 3 words define a background subject 
area, not in itself sufficiently relevant o On the other hand, the relevance 
is apparently much higher when DRIFT occurs which, in this request, is a 
highly selective wordo We noticed a marked tendency for abstracts containing 
words appearing infrequently in the Run 0 output to be assessed relevant o 
It is tempting to apply this model generally and to suppose that these 
words are the more selective, and if weighted would lead to higher precisiono 

Lists '/ere made mechanically for each request of request key- words 
occurring in 80^ or more and in 90^ or more of the output for Run 0, 8C^ 
seemed the better dividing line between the two classes of word® 

The strategy of run 22 is similar to the basic KWS strategy in that the 
key-word stems of a request are compared with those of each abstract, no 
associations being ccnsideredo Words that occurred in less than 8p^ of the 
output for Run 0 contributed 2 to the "coordination value" insteac? of 1 . 
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VI# 2 Discussion of strategies 

VIo2o1 Methods of using word associations 

The associative strategies fall into two main groups corresponding to 
the adopted methods of employing information on word association and 
similarity measures, which are: 

(i) use of word clusters, derived from the word associations, as 
descriptors for indexing and for comparing indexed abstracts 
with requests, and 

(ii) direct use of the word association factors and similarity 
coefficients in the indexing and retrieval processes# 

Using method (i) five different sets of clusters were tried# Though 
derived from the same basic set of word associations, they were produced by 
a variety of methods and they exhibit quite distinct characteristics # For 
example, the clusters in set are mutually exclusive whereas some of the 
other sets contain considerable overlap of words# The number of clusters also 
varies considerably between the s ets# 

Method (ii) avoids the problems and effort involved in generating word 
clusters. It was used in three strategies to provide, by means of comparison, 
some indication of the benefits of clustering# 

VI.2#2 Discussion of our use of word clusters 



Clusters used as descriptors are assigned in the simplest possible ay 
to both documents and requests, one or more word stems in common between a 
document or request and a cluster being the sole requirement# Furthermore 
no explicit system of weighting is used, equal importance being attached to 
all assigned clusters. It would therefore appear that the assumption is 
being made that all the wo 'ds in a particular cluster are inter- substitutable 
in the given texts for retrieval purposes# This is true only in the case of 
the single set of mutually exclusive clusters. The position is not so simple 
in the oe.se of clusters of words which overlap© 

Consider the following situation in which A,B,C, etc#, are words 
arranged in overlapping clusters, all the cluster's containing A being 

(A,B,C,D) 

(A,B,E,P) 

A,B,G) 

(A,B,D,E,H)„ 

An abstract containing any of the above words would have one or more of these 
clusters assigned to it# Notice that if an abstract contained word A or 
B then all four clusters would be assigned# VV)rd D, on the other hand, 
would only lead to the assignment of two of the clusters# Thus, the simple 
use in this way of overlapping clusters provides an in-built system of word 
weighting. Given the occurrence of word A in a search request, words 
appearing in abstracts are weighted as follows by virtue of the above clusters 
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A 4 

B 4 

C 1 

D 2 

E 2 

F 1 

& 1 

H 1 . 

As a practical example of this we have considered, in cluster set C3, all 
those containing the word stem STORo There are 1,178 clusters in C3j having 
a high degree of overlap. The following list shows all the word stems 
appearing with STOR in at least one cluster and the respective weights they 
would receive for a request containing the stem STOR: 



STOR 


14 


MEMOR 


5 


BIT 


13 


SPEED 


5 


DIG-IT 


11 


DECIMAL 


4 


IKPORMAT 


11 


MACHINE 


4 


DRUM 


10 


SYSTEM 


4 


ACCESS 


9 


LOG-IC 


2 


COMPUT 


7 


CRYOTRON 


1 


READ 


7 


HEAD 


1 


TAPE 


7 


PRESET 


1 


READING- 


6 







A set of clusters with little overlap will produce correspondingly less 
refined word weigntingSo 

VIo2.3 Comparison with random superimposed coding 



It is interesting to compare our use of clusters of associated words 
with Mooers* system of Zatocoding or superimposed coding Jo In this 
system key words used to index a document are represented by cutting notches 
in the edges of a card uniquely representing the documento The number of 
positions in which notches can be made is usually in the region of 30 to 50 o 
In order to accommodate a key-word vocabulary probably far in excess of this 
number each key-word is represented not by one particular notch position, but 
by a lonique set of notch positions o Usually about i*our positions are used for 
each word, and these are decided by using random number table So The words in 
a search request are similarly transformed into a set of notch positions and 
documents are retrieved corresponding to cards having notches in th'* desired 
positions. 
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Three observations can be made: 

(i) Since documents and requests are ultimately "described" by sets 
of notch positions, the total set of available notch positions 
functions as a set of descriptors, 

(ii) The transformation of key -words into descriptors is non-unique 

in the sense that different sets of key-words might be transformed 
into the same set of descriptors, and 

(iii) The transformation of key -words into descriptors is completely 
arbitra3?yo 
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This forms a very close parallel to our use of word clusters as 
descr^-ptors, each notch corresponding to a word clustero 
The use of mutually exclusive clusters corresponds in Zatocoding to a 
situation in which each key -word is represented by a single notch positiono 
The two main differences between the use of overlapping clusters and Zato- 
coding are: 

(i) Whereas all key -words are represented by the same number of 

notch positions using Zatocoding, in our system different words 
occur in different numbers of clusters, and 

(ii) In our word cluster systems the relationship between key-words 
and descriptors is not arbitraiyo On the contrary, it is 
carefully contrived so that associated words are clus teredo 

Our method of employing word clusters as descriptors may thus be thought 
of as a generalization of Zatocoding, or superimposed coding, in which the 
coding is associative or probabilistic rather than raiadomo 

VT« 2 o 4 Representation 

Consider a search request containing three key-word stems pi, q^ , ri o 
Let the set of all word stems appearing in at least one cluster containing 
Pi be [pi 5 P2 5 P3j P4j^ in those containing q^ be qs! and in 

those containing be [r^, r2, T3} o The strategies employing un-weighted 

I resentation effectively formulate this request as 

(p 1 +P2 +P 3 +P 4 ) *> (^li ) o (ri +r2 +r 3 ) , 

where and *0* signify the logical operations of ’OR* and ®ML* 

respectivelyo 

We are saying, for example, that an abstract containing Pioq2oT x.. 
better than one containing Pi«qioq2, because the former represertd -‘'1 
three request words whereas the latter represents only twoo ,<evei we also 
imply, for example, that an abstract containing Pi «P3 op4 • ^r2 oTs is no 

better than one containing P3oq2ori, as these represent the same number of 
request words. 

The in-built word weightings produced by using overlapj^i.ng clusters 
vanish when representation is introducedo The occurrence in an abstract of 
an actual request woid contributes no more to the representation value of the 
abstract than the occurrence of another word clustered with a request wordo 

VI o 2 o 5 Direct use of word associations 

A word-connection matrix (Section HI* 10 ) is used to expand the request/ 
abstract texts by addition of all connected words. Retrieval is then 
performed by word comparison, the coordination level being the number of 
common key-words. The words connected vdth a given one are regarded as 
possible substitutes for it for the purpose of retrieval. The first thing we 
tried, therefore, was expanding the abstract texts and comparing them with 
the (unexpanded) requestso The effect of doing this and then retrieving by 
key-word coordination is to accept connected words as substitutes (Rims 15 
and 18)0 

At first sight it does not seem logical to expand the request texts in 
Q 'his way and then retrieve on the basis of key-word coordinationo In this 
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case the maximum possible coordination value would equal the number of words 
in the expanded request and, in effect, the search would be for abstracts 
(or expanded ones) containing all words in the request and all words connected 
with them. However, in the case when the abstract texts are also expanded 
the effect is to produce a system of word weighting similar to that obtained 
using overlapping clusters. If an abstract contains a key -word in a request, 
then the expanded versions of both contain that word and all words connected 
with it, and they all contribute to the coordination value. A word in an 
abstract, not present in a request, may be thought of as having a weight in 
relation to a request word equal to the number of words with which they are 
commonly connectedo This is their contribution to the coordination valueo 
This method was tried in run 23 o 

VIo2.6 Obligatory request words 

It seems generally to be sound practice for a search strategy to accept 
words associated with those in a request, or at least to attribute some weight 
to themo However, in discussing the matter with the requestors we found that 
occasionally a word was used for which the requestor could think of no useful 
substituteo It is quite likely in such a case that the word in question has 
associations with other words which would be misleading in the given contexto 

Words in all requests were considered by their authors and those whose 
presence in retrieved material was felt in this way to be obligatory wer 
underlined. Some strategies were then repeated with a slight modification 
so that abstracts not containing the stems of underlined words were not 
ratrievedo 
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VII 0 I Relevance 



Each abstract was assessed as relevant or irrelevant by the subject 
expert who contributed the original request. The following code was used:- 

3 Relevant, would be likely to contribute 
to a collection of documents answering 
the question, 

2 Irrelevant, would not be likely to do so, 
not worth following up, 

1 Unassessable owing to lack of numerical 
data or the like, 

0 Not yet assessed. 

All abstracts were keypunched omitting numerical or symbolic data, 
equations, formulae, or bibliographic references. This was because, with 
the keypunching available under contract for the first batch of abstracts, it 
was very difficult to check such data to a reasonable accuracy. Further, it 
was hoped that this omission would not greatly affect our examination of a 
system based essentially on key-words, and this has turned out to be so. 

It was strictly for abstracts deficient for this reason that category 
M * was made available. However, out of some 17,000 assessments, about 
2,000 were ’3* while there were no less than 500 M’s. Clearly some 
assessors were using it to indicate a lower degree of relevance and a letter 
was sent out asking for a firm decision wherever possible. 

Finally 83 M *s remained, and in what follows these have been included in 
the *2’s for simplicity’s sake. 

Note that the machine does not offer a relevance judgement. It may 
output an abstract for a certain cut-off K, but not at a lower K-value, 

The assessors on their part used subjective considerations, independently, 
for example, of the number of request words present, 

VIIo2 Precision 



The customer specifies a cut-off value which we denote by K, He asks 
for as near as possible K documents, with, let us say, fewer rather than 
more in the case of a tie. Now 0 and 2K are equidistant from K, If the 
top stratum has 2K or more documents, the strategy is rot sufficiently 
sensitive for the customer’s purpose and gives no output. Thus in any case 
0 ^ output ^ 2K-1 , where 

(i) Output = Total number of Documents Retrieved on a Particular 
Occasion, 

We shall use 

(ii) , Relevant Output =; Number of Documents in Output Judged Relevant 

by User 
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(iii) 



Precision 



Relevant Output 
Output 



If a customer increases the K-value the output will increase or not, 
according to the relative size of the next stratum. If it does, he may get 
more relevant documents, but usually he must expect them to be * thinner on 
the ground* o For the machine can respond only by lowering the acceptable 
standard of matching, and this tends to Icwer precision as just defined. 



To compare two strategies we take a set of requests , so that totals or 
averages may compensate for individual variationso Originally we assumed 
that identical K for both strategies would yield much the same total or 
average output, but this turned out not to be so (below section VH*3) o 
Accordingly the K values were adjusted so that each strategy yielded 50 
documents or nearly on average© Within one strategy K is still held 
constant. 



We then calculate 

(iv) Total Relevant Output over all Requests 

(v) Precision for all Requests* Output Combined 

(vi) Avei age Precision Ratioo 

This is not, however, the whole story© A key-wor*d or key-word stem 
(KWS) match is the simplest, and, involving no associations, remains the 
cheapest to set up and operate© A customer will want to know whether the 
extra cost of associations is worthwhile© Moreover, many documents will score 
highly in both types of run© 

Accordingly, we are particularly interested in the numbers of new 
documents which could not be found by a basic KWS strategy in a reasonably 
sized output© 

We calculate 

(vii) Total New Relevant Output, i.e© not also retrieved in KWS at 
same or related K-value 

(viii^ Precision Ratio of New Output, taRen separately 

(ix) The number of requests which make a positive contribution 
to (i)© 

VII© 3 Operational and evaluation parameters, K and K* 

Here a parameter is simply a variable which we can conveniently control 
in one or more strategies, in order to alter or compare their performance© 

Examine the following figures 



Request Set 


Run 


K 


Relevant 

Abstracts 


Irrelevant 

Abstracts 


Output 


K' 


Precision 


y3 


13(IOTS) 


50 


799 


2443 


322 ^ 


35 


25^0 


93 

93 


9(13T14) 

23(EAR&4) 


50 

50 


972 

871 


38it2 

3441 


4814 

4312 


52 

46 


20fo 

20fo 



These show for three different rims the response when the cut-off K is 
set at 50© Of the three. Run 9 produces most relevant documents© However, 
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the customers have had to scan half as much material again as in Run 13 in 
order to find themo Now, as we have said, the physical quantity of output 
is one of the chief factors in determining the cost and convenience of an 
operational system. Thus while cut-off K is the natural operating 
. Pv arameter a average output is the more appropriate evaluation parameter o 



Reset the K- values till the K* are as near 50 as possible, rather than 
the K. Compare then 



Request Set 


Run 


K 


Relevant 

Abstracts 


Irrelevant 

Abstracts 


Output 


K» 


Precision 


93 


13(KWS) 


71 


991 


3637 


4628 


50 


21^ 


93 


9(13T14) 


49 


966 


3729 


4695 


50 


2 ^% 


93 


23 (EARG4) 


53 


914 


3793 


4707 


51 




Here Run 1 3 


has most relevant 


documents© 


Also, 








If we 


compare strategies 


at the same 


K* then precision is 


proportional 



to the number of relevant documents retrieved, and we need not quote botho 
VJIo4 Recall estimates 



For a given request, let us define 



(i) 



(ii) 



Perfect Recall = Number of Relevant Documents in Collection 
and, given in addition a Strategy and Cut-off, 



Recall Ratio 



Relevant Output 
Perfect Recall 



We have mentioned in Part I the difficulty of estimating perfect recall; 
let us write 



(iii) Known Recall 



Number of Relevant Documents retrieved by at 
least one of the 14 standard strategies 



^ ^ ^ . Relevant Output 

(iv; Known Recall Ratio = — — o 

Known Recall 

Largely manual searches of two types were made to test how meaningful 
these were. 



Virtually Exhaustive Search, 3 request^, (Run 2?) 



The library was scanned as follows: - 



1 o Retain mechanically all documents having one or more words present in 
or associated with the request via the word-word association matrix G-3, and 
print out those not previously assessed© 

2o Mark documents having even the slightest connection with subject matter 
of request© 

3. Send to requestors for thorough assessment© 

Step 1 retained 5^000 documents on average. Step 2 about 200© 
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R elevant Documents found 





Bull 13(KWS) 


Other Standard Runs 


Bun 27 EXHSEA 


Totals 


Request 1 0 


55 


20 


11 


86 


38 


4 


1 




9 


91 


9 


14 


9 


32 




68 


35 


24 


127 



If these figures are at all typical, then our combined runs have retrieved 
of the order of 80^ of the relevant documents in the collection* Clearly 
the figures 11, If, 9 in the fourth column are too small to have been estimated 
by a random sample much smaller than the whole library itself. 

Subject Index Search^ 12 requests^ (Run 26j^ 

Appendix B 1 6 shows a page from the annual subject indexeso The headings 
vary little from year to year, while the entries are based on the abstract as 
a whole but are often rearrangements of words in the title. 12 of the requests 
were searched by their requestors under headings chosen by themselves. Since 
the entries did not give a very full picture the abstracts were found and 
printed out, and some 30fo were chen found to be irrelevant. Of the 1 02 
relevant, 41 7ere known through Run 13 at K* - 50, and a further 19 through the 
other standard runs. Thus 42 were quite new, against 238 already known, 
suggesting that the standard runs accounted for 80fo or more of the totalo 
The following diagram shows the numbers of , relevant docioments involved. 




We have explained our reasons for using K or as our independent 
parameter, and numbers of docioments, or precision for given output, as our 
chief evaluation measure. 



For interest's sake we have added, in the rims tabulated in Appendix B8 , 
three other measures which are worth consideringo Overall Precision and 
Overall Known Recall refer to an imaginaiy request whose numbers of documents 
are the numbers for each request in the particular set added toge there Thus 
Total Relevant and Overall Precision are what we have used alreadyo 



of 



If for fixed K request i 
2 ^ known relevant then 

Overall Precision = 



retrieves x. relevant in 
1 



y^ output out 



Sx. 



1 






Overall Known Recall Ratio 




Zz. 

1 
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1 X. 1 X. 

Average Precision - — 2 — Average Known Recall Ratio = 2 

n ^ ^i 

v/here n is the number of requestso 

With the idea that a strategy yielding x^ = = 0 is not very 

selective we have set precision equal to zero in such a caseo (it happens 
that z^ > 0 for all our 53 requests) o The smaller the value of K the 
greater the chance that yi = 0 and consequently the average precisions 
actually increase for the first few Ko The two recalls, and the two 
precisions after the initial rise in average precision, are not much different 
in size© There does, however, seem to be a systematic difference of a few 
per Cent so that 



Overall Precision Average Precision in general 



o o o (VIIo 5 o 1 ) 



Now 



Overall Precision 







9 



ooo (¥1105.2) 



X. 

that is, it is an average precision in which precision ratio — is given 

^i 

weight y^o The outputs y^ are approximations to K but have a definite 
spread© Aether the output for an individual request is greater or less then 
K is likely to be independent of the request's particular output-precision 
curve o Hence we might expect that requests whose output went above K had a 
lower average precision than those that went beloWo Consequently, in 
(■VIIo5o2) the lower precisions v/ould carry greater weight and this is what 
we have found in (VIIo5»l)* Tn confirmation, we may verify that the effect is 
much less marked in Runs 14 or 28 than in Runs 13 or 25, where the output 
spread is greatero 



The same effect was noticed by Cleverdon [l Volume II, pages 53-55 
for certain of his measures, Vf'hich were calculated by keeping key-word stem 
coordination constant and then totalling or averaging over a set of requestSo 
It would not be surprising if, for a fixed coordination, large outputs 
coi'related with small precision, and this is what Cleverdon 's observation 
implieSo 
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VIII Discussion of Performance Fimres 

Vllloi G-eneral Comments 

Tliere is naturally no single answer to the question ’which is the best 
strategy?* Performances differ from request to request; it may not be 
worth going on to a second strategy, altering the cut-off, or using under- 
lining. Broadly speaking we have in mind the person whose requests are 
batch processed along with many others, so that feedback is likely to be 
limited compared with an interactive system with an on-line terminal o 

We shall use the following terms 
Basic Strategy - Run 1 3 

Standard Strategy - Runs 6, 9, 11 ^ 13 - 23^ fourteen in alio 

A cut-off value of K* ~ 50 is implied lonless the contrary is stated© 
Assessments for runs 24? 25 and 28 are known only where these overlap the 
standard mans, and we write *‘''25, etc© where the assessments are incomplete. 
However, Run 28 is almost fully assessed at K* ^ 50 and we shall not go far 
wrong if we take the unassessed for this K* as irrelevant. We have 
therefore included Run 28 in most of our tables© 

We asked for requests to be formulated in ordinary natural language. 

One or two contained words such as 'information* used in the non-technical 
sense, and which we thought might produce noise in the system© It appeared 
however that say 4 other key-words were sufficient to define a subject area 
so that the presence of such a key -word seemed to have little effect. 

We were not in a position to work on improving or altering the wording 
of requests to see how this affected performance, other than by ’underlining* 
(VIo2©6)© However, we picked out those requests which we felt were less well 
formulated (Appendix B3 )© It turned out that the precision ratios of these 
17 were significantly lower in Run 13 than of the remaining ?6, at the ^ 
level, but we did not feel justified in cancelling them also© 

VIII o 2 Relative Performance of Standard Strategies 

Appendix B9 shows the numbers of relevant abstracts output by each of 
the standard strategies for all 93 requests for a selection of K*© It 
appears that of these 13 or 28 is the best man uniformly as K® varies© This 
is far from being the whole story, however. Some of the runs which do nearly 
as well as Run 13 d.o so because their output is little different. Since 
Run 13 does as well as any, let us assoome that a requestor is likely to make 
it his first try© 

Consider what happens if he goes on to try a second strategy© The 
machine is programmed to suppress all dociaments he has already received and 
presumably scanned© Appendix B9a shows the new docooments output, that is, 
not also output in Run 13o Thus for example at K* ^ 50, Run 9 produced 966 
documents as against 991 documents for Run 13o Only 1 76 of these were new© 
Run 6 produced 753 but 238 of these were new© Again Run 21 produced 902 
documents of which 311 were new and so is better than Run 6 on both counts© 
Consider the run mombers rank ordered by averaging their rajik orders at 

K* 20, 30, 40, 50;- 
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93 Requests: Run Numbers in Averaged Rank Order 

(highest) (lowest) 

Total Relevant Outputs 13 *28 9 22 23 19 21 2011 14181516 1?6 

New Relevant Output: 21 18 15 = 23 61 I =1417 = l6l9 20 22 ^^28 ^ 

There is a weak anticorrelation between these two orders (omitting 1 3 in 
the first set), namely p = -o375 so that, roughly speaking, the nearer a run 
is in performance to Run 13 the fewer new documents it produces p 

It is noticeable that some runs achieve a comparatively high total by 
doing very well for fewer requests. Thus Run 19 produced 21 8 new documents, 
but only 4^. out of 93 requests contributed to this totalo This coverage or 
number of productive requests is noted in brackets in the appendices where it 
is knowno The coverages for one relevant document differ little,"^ but those 
for new relevant documents are perhaps worth putting in average rank order: - 

93 Requests: Run Numbers in Rank order 

(highest) (lowest) 

Coverage for New Documents: 6 23 11 I8l5 2l 1416 = 20 1719 9 *28 22 

This seems to correlate to a fair extent with the numbers of new documents 
retrieved (P= - 80 )o 

The numbers of new documents are an important test of an association 
strategy's performance^ The motive for going on to a second strategy is much 
stronger when Run 13 produces only a few documentso It does not necessarily 
follow that in such a case other runs are likely to produce even fewer© 34 
requests, 37 per cent of our set, produced only 0 to 4 relevant documents in 
Run 13 at 50, (Appendix BIO )o Even if we omit the less well formulated 

requests^ there are still 13 for which we need output of 250 or more if we 
are to get further relevant documents© New documents however are often 
retrieved for this set by association or descriptor 2 runs at low Ko 

If we take the standard runs at K® 50 and then select the 34 requests, 
we get the figures in Appendix B1 1 and the following rank orders:- 

34 Requests: Run Numbers in Rank Order 

(most) (fewest) 

Total Relevant Output: 15 1 8 11 9 23 *28 21 20 14 6 17 I 6 19 *25 22 13 

New Relevant Outputs 21 15 1 8 23 11 9 14 *28 6 20 17 l6l9 *25 22 

Coverage for New 1811 23 6l5 21 9 I4l7l6 20 *28 1 9 22 

Documents: 



'^'Compare, however; - 

Strategy 28 11 = 9 23 13 6 = 1 5 = 1 8 = 21 = 22 20 14 = 1 9 1 7 1 6 

Numbers of Requests 65 64 64 63 59 58 58 58 58 58 57 55 55 53 51 

with 5 or more 
Q relevant documents 
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When we select requests at random we might expect the to remain 

roughly the same, and this holds here for the 34 requests for all runs except 
13 anti 19 (K' ^ 33 and 35 respectively ) o This drop from K' 50 makes us 
ask, in choosing those requests with low relevant output have we simply chosen 
those with lower than average total output? If we raise the K®' to co\inter- 
balance Run 13^S low sensitivity, thus. 



Request Set 


Run 


K Relevant 


Irrelevant 


Output 


K' 


^ Precision 


34 


13 


87 101 


1372 


1672 


49ol8 


60 O 4 



... (viii.2,1) 

we still find Run 13 below the other runs, except Run 22, and so the poorer 
performance of the 34 requests in Run 13 must be due in great measure to 
semantic considerationso In fact, no single run at K* 50 is able to cover 
more than 65 out of the 93 requests at the level of 5 relevant documents, 
while there are 83 requests for which 5 or more are known o 

We may use ^ as our measure of sensitivity (Vo7)o Runs may be rank 

ordered according to the cut-off values K needed to yield a given average out- 
put (Appendix B5), a high K giving low sensitivity o We deduce the 
following orderss 



93 Requests; Run numbers in order of sensitivity 

(highest) (lowest) 



K' 20 


6=11 


23 


9 = 


28 


= 14 21 


= 20 


17 


15 


= 16 22 


= 13 


19 


= 25 18 


K* - 30 


6 = 11 


9 


28 


23 


14 21 


22 


15 


18 


= 13 19 


= 17 


= 16 


20 25 


K' - 40 


6 = 11 


9 


23 


28 


14 21 


22 


18 


15 


19 20 


13 


17 


25 16 


K' - 50 


6=9 


11 


28 


23 


14 21 


= 22 


18 


19 


13 15 


20 


17 


16 25 


Average rank order 


6 = 11 


9 


28 = 


23 


14 21 


22 


13 


18 


13 20 


19 


17 


16 25 



Note that sensitivity is independent of any relevance judgments. 

VIIIo3 Run 13 with a Second Strategy compared with Run 1 3 at Higher Cut-Off 

We have discussed above the numbers of new documents produced by our 
standard runs* The best of these was Rim 21 o Let us unite the document sets 
produced in runs 13 and 21, writing U for set union, thus:- 



Request Set 


Run 


K 


Rel evant 


Irrelevant 


Output 


K' 


% Precision 


93 


13 U 21 


71,62 


1302 


6006 


7308 


780 58 


17o82 


93 


13 


113-115 


1185 


5880 


7065 


75o97 


16o77 


93 


13 


116-117 


1204 


61 85 


7389 


79o45 


16o29 



We have included the results for Run 13 which are nearest in total outputo 
We do a little better by using 13 U 21 than by increasing the output for 
Run 13 aloneo Thus the two strategies together ai'e better than either of them 
singly* Although we have not distinguished documents so long as they are 
reDcvant, the requestor may appreciate the increased variety brought in by the 
association runto 

The improvement is even more marked for the 34 set* M3 H15* indicates 
documents in Run 15 ^ut not in Run 13« 
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Request Set 


Run 


K 


Relevant 


Irrelevant 


Output 


K* 


'fa Precision 


34 


13 


71 


79 


1058 


1137 


33o24f 


6o95 


34 


_ 15 


73 


141 


1464 


1605 


47o20 


8,79 


34 


13 0 15 


71,73 


77 


958 


1035 


30.45 


lakh 


34 


13 U 15 


71,73 


156 


2016 


2172 


63 088 


7o18 


34 


13 


110 


123 


19i|2 


2065 


60o74 


5o96 


34 


13 


111 


124 


2148 


2272 


66082 


5o46 


Note that when 


two runs, 


at say K’ - 50 , are 


united 


the new 


average 


output per request will be something less than 1 00 , 


the sum 


of their K*, 


and vary somewhat between different pairs of runs. 


according to their overlap o 


Thus greater total relevant output may 


not correspond strictly with greater 


precisioHo 


We may compare the 


1 following figures with 13 U 


21 aboveo 


Request Set 


Ron 


K 


Re event 


Irrelevant 


Output 


K' 


fa Precision 


93 


13 U 19 


71,66 


1 2 09 


5219 


6428 


69.12 


18.81 


93 


13 


1 04-1 09 


1158 


5082 


6240 


67.1 0 


1 8o56 


93 


13 


110 


11 66 


5276 


6U2 


69.27 


18„10 



VTIIo4 C- enerality of a Request 



We define 



G-enerality of a Request 



Number of Relevant Documents in Collection 
Total Number of Documents in Collection 



0.0 (VIII. 4ol) 



It may be thought of as the precision should the whole library be output o 
Requests with low generality are in some sense likely to be harder to answer o 
Some previous workers who have had the whole of their document collection 
assessed have therefore studied generality as an additional variable. If we 
use total known relevant in the numerator of (VIII.4.l), the generality of 
our requests ranges from o0086^ to We may divide our set of requests 

into four groups of increasing generalityo 

Known Relevant ^ G-enerali*ty Number of Requests 



&EN 1 


0-8 


„000-o069 


26 


G-EN 2 


9-17 


0 078-0 15 


21 


&EN 3 


18-29 


0I6 -.25 


23 


&EN 4 


34-84 


o26 -o73 


23 



See Appendix B 12 .for the corresponding perfoimance figureso The figures 
for lowest generality are perhaps too small from which to draw any strong 
conclusions, but the other three show a rank order very similar to those for 
all 93 requests together (A/1IIo2)o 



Rank Order of Run Numbers, K* - 50 ^ for Relevant Documents 



Request 

O 

ERIC 



(most) 

G-EN 1 11 22 6 18 

G-EN 2 9 1 3 21 1 9 

&EN 3 22 1 3 1 1 9 

&EN 4 1 3 9 1 9 23 



13 


9 : 


0 

(M 

11 


17 


22 


11 


15 


18 


= 19 


21 


23 


15 


14 


21 


20 


18 






65 



(fewest) 

16 23 15 19 21 14 
= 23 20 6 14 16 17 
14 20 6 18 17 16 
16 17 22 15 11 6 
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New Relevant Documents, 


not 


in 


Run 1 


I3 






Request Set UM 


1 


11 


18 


6 


21 


15 


17 16 


= 23 


14 


=; 20 


9 


22 


'i9 


GEN 


2 


21 


15 


18 


11 


23 


= 20 1 9 


9 


14 


6 


16 = 


: 17 


22 


GEN 


3 


11 


= 15 


23 


21 


6 


14 18 


9 


19 


20 


16 = 


= 17 


22 


GEN 


4 


18 


15 


21 


14 


19 


1 6 1 7 


11 


23 


6 


20 


9 


22 



VIIIo5 Underlining (55 requests ) 

Underlining (VI.2o6) was first tried with descriptor Run 14 to give it 
additional precision, making Run 21 o Later it was tried v/ith Runs 13 £«J^d 11 
after the standard runs had been assessedo The strategies were first applied 
to all 93 requests at K* ^ 30, and then the 55 requests selectedo The 
figures in Appendix B13 include the resultant effective K®, whdch do not 
differ enough from 30 to affect the argumento 

(i) One Underlined Strategy 

Referring to Appendix B1 3 , we see that underlining slightly 
improves the performance of each of the three runs so that 25 
(U13) is the best of alio But U1 3 has quite a striking decrease 
in sensitivity over Run 13^ needing, for example, K = 8l to 
produce K* ~ 50 o 

(ii) Two Strategies in Succession, One Underlined 

Appendix B13 shows that it is best to combine with Run 25 not 
Run 1 3 or nother underlined strategy (Run 21 ) , but an association 
or descriptor run, with Run 23 heading the listo Thus:- 



Request Set 


Run 


K 


Relevant 


Irrelevant 


Output 


K‘ 


% Precision 


55 


_ 25 


29 


356 


660 


1 01 6 


18 


35 


55 


25 n 23 




155 


629 


784 




20 


55 


25 U 23 




511 


1289 


1800 




28 


55 


25 


58 


>511 




1853 


34 


>29 



It would seem that Run 23 is the best to combine with Run 25, 
but that we could get similar performanc ^ on average out of Rian 
25 alone with increased cut-offo What about the requests, 
although underlined, for which Run 25 does badly? If a request 
does badly in Run 25 = U13, it may still be worth tiying Run 13 
but an association or descriptor run is more likely to break new 
groundo 

However, in underlining, we are limiting oursv^lves to a particular 
subset of abstracts and it is for the requestor to decide whether 
this is desirable in any particular caseo 
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IX SiiTnmary and ConolusionB 
IX* 1 Evaluation c riteria 

Our aim was to find what improvements, if any, could be made upon the 
performance of retrieval by matcliing key-word stems by the use of word 
associations and clusters o Thus run 13 (KWS) is our standard of comparison. 

In assessing different strategies v/e consider three measures to be particularly 
useful: 

(i) The total number of relevant documents retrieved for a set of 
requests in a given quantity of output* 

(ii) The sens it ivity of a strategy, that is, its ability to output a 
number o. aments uniformly close , to that requested* 

(iii) The coverage of a strategy, that is, the number of requests for 
which it retrieves some minimum number of relevant documentSo 

Por associative strategies (i) and (iii) are calculated using total numbers 
of relevant documents, and also using numbers of new relevant documents with 
respect to the output of run 13* 

As far as we are aware the concept of the sensitivity of a strategy has 
not previously been discussed in the literature”^ ^ Measure it (Vo7) ^ K’A> 
the output per request averaged over a set of requests divided by the desired 
output o In changing the strategy we find that as the relative spread of the 
outputs for different requests about their average increases so K^/K decreases. 
With very lov/ spread approaches unity* Both these effects depend on the 

number of strata into which the strategy partitions the output. Low 
sensitivity implies that the performance for some requests may suffer by very 
little output being produced* As regards (i) v/e have offset the effect of 
differing sensitivity by testing each strategy at the same K* rather than at 
the same K. 

The recall performance of the various strategies may be assessed in terms 
of the total numbers of relevant documents they retrieve in relation to the 
total number of known relevant* Results of virtually exhaustive searches on 
three requests and of subject index searches on 12 (section 'VIIo4) suggest 
that the numbers of known relevant (total and for individual requests) 
represent about 8C^ of all relevant documents in the collection * 

IX* 2 Performance of key -word stem searches 

In relation to results yielded by our various associative strategies it 
must be concluded that retrieval by the simple means of comparing key -word 
stems (run 13) provides a very good level of performance* The total of 991 
relevant documents retrieved for the full set of 93 search requests was 
matched only by run 28 which uses a refinement of the word stem search** 

This represents a recall of about 40^ of all relevant docximents in the 
collection* Between them, all the strategies described (including run 13) 
produced 2,020 relevant documents* 



ERIC 



*Qhi<^ superiority of simple key-word stem searchinj when judged purely on the 
ba^sis of total numbers of relevant documents retrieved closely parallels 
some of the findings of Cleverdon et al Ll5_f« Our use of key-word stems is 

very similar to their language consisting of single terms with confounding 
® .word forms, which they found to perform better than others they tested* 
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The weakness of key-word stem retrieval is its very low sensitivity 
(K*/K » 0o7)o A five word request, for example, splits the collection into 
■ at most five strata whether it contains 10,000 or 100,000 documents* There is 
a large spread in quantity of output with a marked tendency towards lower 
output than desired* Thus, in run 13 it was necessary to retrieve as close 
as possible to 71 documents for each request (K = 71 ) in order to produce an 
average output of 50 documents per request (K* = 50)o Even at this output 
level run 13 produced little or no relevant material for many requestso 

In run 28 the key-word stem strategy is refined by ordering the documents 
in each of its strata according to their coordination values in run 14o This 
subdivision of the strata produces excellent sensitivity (K^K = 0*97) o The 
total number of documents retrieved for 93 requests is the same as in run 13« 
However, this reflects a balance between 155 new relevant documents for 
requests for which run 13 produced low output and 155 fewer for a smaller 
number of requests for which run 13 produced high outputo Appendix B11 
s umm arizes results for a subset of 34 requests for each of which run 13 yielded 
fewer than five relevant answers* For these, run 28 produced 130 relevant 
documents compared with 79 in run 13o 

IXo3 Advantages of associative strategies 

Our strategies employ word associations in one of two ways: 

(i) Word clusters formed from them are used as descriptors which are 
assigned to abstracts and requests. Retrieval is then based upon 
comparison of clusters* 

(ii) Abstracts alone, or abstracts and requests, are expanded by adding 
associated words* Word stems in the expanded texts are then 
compared when retrieving* 

We have found strategies of both kinds with good levels of perfonnance a^^d 
naving three things to recommend them: 

(a) Very good sensitivity > 0*9)° 

(b) Fairly good recall of new relevant material* Although they retrieve 
fewer relevant documents in total than run 13j the best strategies 
add to the number of relevant documents found in that run by about 
3ofo. 

(c) Particularly good recall of new relevant material for requests 
for which a word stem search yields little or no relevant output* 

For the subset of 34 such requests the best strategies increase the 
number of relevant documents found in run 1 3 by about 75fo (see 
Appendices BIO and B1l)* This improvement is accounted for partly 
by the better sensitivity of the associative strategies and the 
fa.ct that a large proportion of the subset of 34 requests obtained 
low total output in run 13o However, in producing the figures in 
table W!II^2.1 we have compensated for this by considering 

a cut-off level which raises the average output of these requests 
to our standard of approximately $0* The figures show the improved 
performance of a word stem search at this output level* The best 
associative strategies are still able to improve upon these new 
figures by about l^Ofoo 

Considered individually, none of these improvements is spectacular and 
it would not be sensible to recommend the use of one or more associative 
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stx’ategies rather than performing a simple word stem search. However, the 
combination of these advantages together with slightly improved coverage 
provided by most of the associative strategies does make their use well worth 
considering seriously. We think their main value lies in their use to 
supplement a key-word stem search when the performance of the latter is found 
unacceptable or when better sensitivity is required. 

One most interesting observation applies generally to our associative 
strategies. We have found a marked negative correlation between the amount of 
new relevant output (relative to the results of the word stem search of r\in 13) 
from strategies and the total relevant output they produce. In other words, 
associative strategies providing the highest relevant output tend to produce 
largely the same relevant documents as run 13^ those providing a good selection 
of new relevant material produce more meagre total numbers of relevant 
dociomentso 

IX.4 Best associative strategies 

Assxamiug that they would be used along with a key-word stem search, our 
final assessment of the associative strategies is based upon their performance 
for the subset of 34 requests for each of which Eun 13 produced fewer than five 
relevant documents. Our 14 main strategies are ranked below according to the 
three criteria listed in section IX. 1 . 
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s-^rategies 11 and 23 show a fairly balanced performance, appearing within the 
top six of each rank ordering. We regard these as the best of the associative 
strategies we have tried. Strategy 11 uses a set of 1,178 highly overlapping 
word groups as descriptors with proportional representation (VI.1a6)o In 
strategy 23 key-word stems are compared after expanding both requests and. 
abstracts by adding words which, according to connection matrix G4 (lllolO), 
are associated with the original words. 

IXo 5 Effects of cluster overlap and representation 

One important feature of clusters is their degree of overlap, and it is 
closely related to the number of words that do not become isolated in the 
clustering process and the number and size of the clusters themselves. In 
general, we find that new relevant output, coverage and sensitivity, 
particularly the latter, are all improved by. increasing the overlap of clusters 
used as descriptors. By contrast, non-overlapping clusters (e.g. strategy 16) 
necessarily lead to lower sensitivity even than key-word stem retrieval. Use 



ERIC 



^In comparing these totals with those for the 93 requests, note that 5 or more 
relevant documents are known to exist for only 24 out of the 34 requests. 
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of representation with clusters (VI,1,5) further improves these measures 
provided there is a sufficient degree of overlap o Proportional representation 
(VIo1 o6) performs even bettero 

IXo6 Effect of obligatory request words 

The results produced by allowing people to make some of their request 
words obligatoiy (indicated by underlining) in retrieved abstracts suggest 
that, used with restraint, this can be a useful precision device^ The 
sensitivity of a strategy is, however, loweredo 

IXo7 Effect of request generality 

The genera.lity of our requests (ioeo ratio of number of documents in 
collection relevant to a request to total number in collection) ranges from 
0o008^ to 0o7^* These figures, however, are based upon numbers of known 
relevant documents, estimated to be 80fo of the totalso The true generality 
range is therefore likely to be 0o0l^ to 0o^^foo The 93 requests were 
arranged into four subsets of roughly equal size each covering a different 
range of generalities* The performance of different strategies was studied 
for these subsets of requests (VIIIo^ socid Appendix B12)o There is no 
evidence of significant performance differences for the different generality 
ranges, nor of significantly different strategy rankings according to their 
performance for these classes of requesto 

IXo8 Choice of strategy 



If it is desired to use only one strategy then a simple key -word stem 
search is the best when its low sensitivity is acceptable, and is certainly 
cheaper to implement than any associative strategy. When better sensitivity 
is essential, for better control of quantity of output, then a refined key- 
word stem search, such as run 28*^, should be employed. This removes the 
likelihood of some requests receiving little or no output and, hence, little 
or no relevant output. 

If better recall and sensitivity are required than it, alone, can provide 
a key -word stem search should be followed by an associative strategy© Of the 
ones we have tried strategy 23 ^ making direct use of a set of word associations, 
is undoubtedly the best choice. Pun 28, mentioned above, is too nearly 
related to Pun 13 to be so useful as a second strategy© Strategy 11, though 
yielding a similar level of performance, would be a costlier choice requiring, 
as it does, the generation of a set of word clusters. We would say, more 
generally, that, of the strategies we have tested, those involving direct use 
of word associations perform comparably with, and are therefore preferable to, 
those employing word clusters. 

If a key -word stem search is found not to be very productive, greater 
improvement in recall can be expected by proceeding with an associative 
strategy than by changing the cut-off level in the word stem search to yield 
the corresponding combined output. 

For some of the factors affecting the cost of the various strategies, 
see Appendix b6. 




*Any associative strategy can be used to subdivide the coordination strata of 
the word stem search. The strategy of run 23 is probably the simplest and 
best choice. 
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I.Xo9 Application to on-line 3 interactive systems 

Some of the teclmiques we have studied could be employed to improve the 
performance of retrieval systems working in an on-line, interactive modeo 
Words and word clusters statistically associated with those in a request or 
in an already retrieved relevant document could be displayedo The searcher 
could then decide which associated words are useful in the context of his own 
request, and how to use them to broaden or modify his re quest o It seems 
likely that word associations could lead to greater improvements in retrieval 
performance when tailored in this way to a searcher's needs than when 
employed in a non-interactive mode. 
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APPENDIX A2 



4-Zone Card Code 

The table below shows the code used when punching the texts of abstracts 
on cards for input to the computer. Each character, numeric or alphabetic, 
is punched in a separate column of a card. Each numeral in the range 0-9 
is coded by a single hole punched in the appropriate row. Each alphabetic 
character is coded by one hole in one of the rows Y,X, 0 and another in 
one of the remain^g rows. 
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APPENDIX A3 

Letter Prequencies within Abstracts 

The program used to read the abstracts from punched cards, perform the 
dictionary look-up, etc., also counted the number of o( iurrences of each 
letter of the alphabet and the number of spaces. All words appearing in the 
texts, not only dictionary words, were included in this frequency count. 

The cumulative frequencies obtained from 1i,57l abstracts appear below in 
descending order. The average word length, as computed from these frequencies, 
is 5o5 letters. 
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AP PENDIX A4 ( CONTINUED ) 

WORD-PAIR FREQUENCY DISTIBUTIONS 

1 BATCH = 1536 ABSTRACTS 
P = FREQUENCY « NUMBER OP ABSTRACTS 



P(F) = NUMBER OP WORD-PAIRS WITH FREQUENCY P 
(OMITTING REPETITIONS WITHIN AN ABSTRACT) 



BATCH 


1 


BATCHES 


1 -2 




BATCHES 


1-4 




P 


P(F) 


P 


P(F) 


P 


P(P) 


F 


P(F) 


0 


443121 


0 


410066 


0 


355154 


71 


6 


1 


33778 


1 


53356 


1 


70954 


72 


7 


2 


9325 


2 


1 6297 


2 


27141 


73 


1 1 


3 


3461 


3 


7062 


3 


13663 


74 


3 


U 


17BI 


4 


3801 


4 


8130 


75 


10 


5 


942 


5 


2376 


5 


5291 


76 


3 


6 


609 


6 


1540 


6 


3712 


77 


1 


7 


336 


7 


1057 


7 


2654 


78 


9 


p 


271 


6 


768 


8 


2066 


79 


1 


9 


206 


9 


555 


9 


1 581 


80 


4 


10 


138 


10 


428 


10 


1272 


81 


4 


11 


100 


1 1 


339 


11 


1039 


82 


4 


12 


82 


12 


256 


12 


320 


83 


3 


13 


57 


13 


216 


13 


706 


84 


4 


1 n 


51 


14 


164 


14 


596 


85 


1 


15 


46 


15 


157 


15 


462 


86 


3 


16 


31 


16 


139 


16 


430 


87 


1 


17 


15 


17 


104 


17 


321 


68 


6 


13 


21 


16 


103 


18 


307 


69 


6 


19 


16 


19 


79 


19 


294 


91 


5 


20 


15 


20 


65 


20 


277 


92 


2 


21 


16 


21 


62 


21 


213 


93 


2 


22 


8 


22 


49 


22 


214 


95 


2 


23 


15 


23 


38 


23 


191 


96 


2 


2^4 


9 


24 


35 


24 


133 


97 


5 


2b 


9 


25 


36 


25 


139 


96 


2 


26 


5 


26 


27 


26 


121 


99 


2 


23 


7 


27 


22 


27 


108 


100 


3 


29 


3 


28 


20 


28 


90 


101 


1 


30 


3 


29 


30 


29 


101 


102 


3 


31 


2 


30 


16 


30 


95 


103 


1 


32 


3 


31 


15 


31 


73 


104 


2 


33 


2 


32 


13 


32 


69 


105 


1 


3b 


2 


33 


19 


33 


64 


106 


2 


36 


1 


34 


12 


34 


46 


loy 


2 


33 


1 




12 


35 


62 


108 


1 


39 


3 


36 


15 


36 


53 


109 


2 


in 


2 


37 


3 


37 


47 


1 10 


3 


42 


1 


38 


7 


38 


35 


1 1 1 


3 


43 


2 


39 


13 


39 


36 


112 


1 


46 


1 


40 


9 


40 


43 


115 


1 


47 


1 


41 


3 


4l 


45 


1 16 


1 


43 


1 


42 


7 


42 


36 


117 


2 


119 


1 


43 


7 


43 


26 


118 


1 






44 


5 


44 


27 


119 


1 






45 


6 


45 


29 


122 


1 






46 


4 


46 


23 


124 


1 






47 


7 


47 


25 


126 


1 






48 


2 


48 


23 


129 


1 






49 


4 


49 


22 


131 


1 






50 


4 


50 


17 


132 


1 






51 


2 


51 


15 


133 


1 






52 


4 


52 


20 


135 


1 






54 


3 


53 


21 


138 


1 






55 


4 


54 


13 


142 


2 






56 


3 


55 


20 


144 


1 






58 


1 


56 


1 1 


145 


1 






59 


2 


57 


11 


149 


1 






60 


1 


58 


13 


159 


1 






61 


1 


59 


10 


1 65 


1 






62 


3 


60 


16 


166 


1 






63 


3 


61 


11 


168 


1 






64 


1 


62 


7 


175 


2 






65 


2 


63 


14 


183 


1 






66 


1 


64 


9 


190 


1 






67 


1 


65 


8 


191 


1 






68 


1 


66 


7 


1 92 


1 






71 


3 


67 


7 


201 


1 






72 


1 


68 


7 


207 


1 






74 


1 


69 


8 


219 










1 

1 


70 


6 


381 


1 






82 


1 

1 














03 


1 














84 


3 














85 


1 














90 


1 














100 


1 














108 


1 














213 


1 










0 
















ERIC 






83 

























BATCHES 1-8 



p 


P(P) 


P 


P(F) 




F 


P(F) 


F 


0 


296779 


63 


39 




126 


6 


195 


1 


78712 


64 


50 




127 


4 


196 


2 


36475 


65 


29 




128 


6 


198 


3 


20741 


66 


37 




129 


3 


199 


U 


13487 


67 


46 




130 


5 


200 


5 


9396 


68 


41 




131 


5 


201 


6 


6941 


69 


28 




132 


1 


202 


7 


5197 


70 


29 




133 


2 


203 


f; 


3967 


71 


38 




134 


6 


204 


9 


3313 


72 


28 




135 


3 


206 


10 


2668 


73 


35 




136 


3 


208 


1 1 


2231 


74 


30 




137 


7 


210 


12 


1931 


75 


25 




138 


4 


21 1 


13 


1 61 3 


76 


27 




139 


1 


213 


U 


1445 


77 


21 




140 


9 


214 


15 


1252 


78 


18 




141 


3 


215 


16 


1069 


79 


21 




142 


3 


21 9 


17 


923 


80 


7 




143 


2 


220 


18 


614 


81 


24 




1 44 


5 


221 


19 


742 


62 


12 




145 


7 


222 


20 


666 


63 


18 




146 


2 


223 


21 


625 


84 


15 




147 


4 


226 


22 


554 


65 


16 




148 


3 


225 


23 


532 


86 


24 




149 


3 


231 


2i4 


440 


87 


22 




150 


2 


232 


25 


432 


86 


15 




151 


3 


235 


26 


373 


89 


17 




152 


4 


237 


27 


365 


90 


15 




153 


3 


239 


28 


292 


91 


19 




1 54 


3 


240 


29 


332 


92 


16 




155 


2 


242 


30 


242 


93 


18 




1 56 


4 


243 


31 


272 


94 


12 




157 


2 


244 


32 


245 


95 


10 




1 58 


5 


245 


33 


210 


96 


15 




159 


2 


247 


3^ 


201 


97 


13 




160 


3 


254 


35 


180 


98 


10 




161 


8 


259 


36 


186 


99 


6 




162 


2 


263 


37 


156 


100 


13 




163 


1 


267 


38 


147 


101 


9 




164 


4 


274 


39 


1 65 


102 


1 1 




1 65 


4 


278 


^40 


140 


103 


14 




166 


2 


287 


in 


133 


104 


9 




167 


4 


268 


i42 


121 


105 


13 




166 


2 


297 


i43 


90 


106 


14 




1 69 


2 


304 


i4i4 


120 


107 


1 1 




170 


2 


305 




83 


108 


19 




171 


2 


309 


ii6 


97 


109 


10 




172 


1 


315 


ii7 


92 


1 10 


10 




173 


3 


321 


iie 


76 


1 1 1 


10 




174 


2 


322 


il9 


60 


112 


6 




175 


1 


324 


50 


75 


113 


7 




176 


3 


330 


51 


62 


114 


12 




177 


4 


351 


52 


77 


115 


7 




178 


3 


359 


53 


f! 


116 


5 




179 


2 


366 


5^ 


64 


117 


n 




163 


2 


382 


55 


71 


118 


6 




164 


2 


385 


56 


63 


119 


3 




185 


3 


397 


57 


54 


120 


3 




187 


2 


399 


58 


48 


121 


9 




188 


2 


449 


59 


44 


122 


6 




189 


1 


461 


6o 


56 


123 


4 




191 


1 


483 


61 

62 

i 


45 

46 


124 

125 


4 

6 




194 


2 


2512 


} 

o 
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- 
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roro-*-*-*-*-*-*-*-* — oorou) 
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774 
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7 1 
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70 

86 

1 2 1 
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74 1 
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BINARY WORD CONNECTION MATRIX G3 



ACCELER A 


1 3 


1 3 


ACOUSTIC 




6 08 
9 25 


ASYMPTOT 

electrom 


1 4 


1 4 


Perpend i 




48 1 


CONDUCT 




9 83 


C YL 1 ND 
WEDGE 


1 5 


3 


WAVE 




1 5 


ACTIVITY 




1 1 2 
4 5 4 


CORREL At 




7 00 


St AT 1 ST 1 
SUNSPOT 


1 6 


1 6 


Cycle 




1 49 


Year 


1 7 


1 7 


ad 1 A8 AT 1 




2 1 


A P P R 0 X 1 M 




2 60 


Passage 




460 


LEVEL 




6 93 


MASER 




755 


ADM 1 TT an 


1 8 


1 8 


1 MPEDANC 




2 05 


REPRESEN 




597 


terminal 




880 


NETWORK 

Pass 1 V 


1 9 


1 9 


M ATR 1 




1 1 2 


ADVANTAg 




257 
9 05 


CONVENT 1 

Operatic 


2 0 


1 


TR 1 ODE 




20 


RELAY 




1 4 1 


Stab 




2 7 1 


AFTERNOO 




5 1 3 
538 


midnight 




6 06 


NONSE ASo 




643 


DAYTIME 




699 


0 1 URNAL 




7 00 


median 




775 


CYCLE 

LAYER 


2 1 


1 7 


alphanum 




2 1 
1 59 


CH AR ACTe 




1 63 


ALTERNAt 




236 
4 1 6 


SOLVE 




580 


AMPL 1 T UD 




672 

8 t 3 


analogue 




8 50 
956 


analyser 




963 


M ATHEMAt 
C OMPUT 


22 


22 


analogue 




4 2 
8 7 


analyser 




1 06 


D 1 FFEREN 




1 59 


an I SOTRO 




2 4 0 
5 2 4 


0 1 electr 
ELECTROM 


23 


2 3 


homogene 


52 0 


1 SOTROP 




8 3 1 


PERMEA8 




8 6 7 


MED I UM 




8 9 3 


TENSOR 




997 


p 1 eld 

MEDIA 


24 


2 4 


AXES 




364 




ocr 


4 4 1 



APP AR ATU 


24 


632 


SEARCH 




85 4 


SCAN 


25 


2 5 


APPARENT 
V 1 RTUAL 


26 


2 6 


LUN 




900 


AD 1 AB AT I 


2 7 


27 


APPROX 1 M 




95 4 


GAUSSIAN 

SEGMENT 


28 


28 


EXPAN 




6 0 


ARGUMENT 




497 
5 4 3 


1 NTERPL A 




9 1 4 


ART 1 F 1 C 1 


29 


2 9 


ATMOSPHE 




1 0 8 


S ATELL 1 T 




4 3 3 


SPUTNIK 




49 1 


EARTH 




5 7 2 


OR 8 1 T 




7 0 4 


ASSOC 1 AT 




75 4 
83 1 


0UT8URST 




87 4 


RECORD 




9 1 5 


HISS 


3 0 


3 0 


ASYMMETR 




356 


GAUSSIAN 




63 0 


RESPONSE 




977 


PASS 


3 1 


3 1 


ACOUSTIC 




5 4 4 


ASYMPTOT 




7 2 8 


INTEGRAL 

SOLUTION 


3 2 


32 


CONVEX 




33 


finite 




1 8 3 


SCALAR 




1 9 0 


UNIQUE 




201 


EXACT 




3 1 4 


EXPAN 




345 


R EVOL 




4 77 


art 1 F 1 Cl 




63 0 
6 8 9 


ATMOSPHE 




869 


1 SOTHERM 
L 1 GH TN 1 N 


3 3 


3 2 


R AD 1 OSON 




33 


orbital 




1 83 


OXYGEN 




1 9 0 


CLOUD 




2 03 


UPPER 




3 26 


DRAG 




4 7 7 


WIND 




636 


A 1 R 




922 


ATT ACHMe 


34 


3 4 


COEFF 1 C 1 




2 1 6 


electron 




8 7 9 


F OR M A T 1 0 




9 8 7 


1 SOTHERM 




999 


RECOMB 1 N 
D 1 FFUS 


3 5 


3 5 


ATTENUAt 

DESIGN 


36 


3 8 0 
3 6 


SAND 
F 1 LT 


3 7 


37 


LOSS 




3 2 8 


D B 




4 8 2 


AUTOMAT 1 


3 8 


3 8 


DEC 1 UAL 


7 0 


READING 




23 5 



SYSTEM 

DRUM 

A V AL ANCH 

8 ACKGROU 
NO I S 

8 ACKWARD 
WAVE 



8 AL ANCED 
C 0 N V E R S I 
8 AL ANC 
H E A T E 
PULL 

8 ANDW I OT 
FREQUENC 
PRODUCT 
AMPL I F 
NARROW 
F I GUR 
OPT I M 
8 AND 
GAIN 
PUMP 

8 I D I RECT 
CONTROL 
S W I TCH 
GAT 

8 I FURC AT 

HEIGHT 

LAYER 

8 I ST ABLE 
SLOCK I NG 
M0N0STA8 
mult I V I 8 
OPERATIC 
TR ANS I ST 
CIRCUIT 
TRIGGER 
S W I TCH 
DIODE 

flip 

8 I STABLE 

SLOCKING 

M0N0STA8 

MUL T I V I 8 

OSC I LL AT 

WAVEFORM 

TRIGGER 

TIMING 

RISE 

B REMSSTR 
PHOTOEL E 
HIGH 
RAY 
X 

CALCULAT 
E V ALUAT 

CAL I 8R At 

CeNTlMET 
w aveleng 
V I S I 8LE 

CERENKOV 
D I ELECTR 
R AD I AT I 0 



57 1 

7 

3 9 

78 1 

8 1 6 

4 0 

6 06 

6 4 7 

69 4 

93 1 

967 

4 1 

22 

42 

24 0 

286 

58 0 

43 

742 

4 4 

8 7 

1 2 1 

1 2 3 

246 

3 1 6 

40 1 

4 06 

4 1 3 

6 2 t 

725 

876 

4 5 

33 7 

46 

4 7 

60 1 

4 8 

^ 0 1 

30 1 

4 b 1 

4 9 1 

49 

3 48 

869 

50 

65 

3 08 

8 1 6 

8 2 5 

5 1 

52 

1 9 1 

2 4 0 

5 1 9 

58 0 

979 

53 

3 4 1 

54 

1 9 4 

339 

55 

377 

1 09 

673 

5 6 



TRANSFOR 



moving 


56 


3 1 2 


ALPHANUM 


57 


57 


CHAR ACTE 
SELEN 


58 


58 


WAFER 




1 9 1 


circular 

SCALAR 


59 


8 29 
5 


VECTOR 




59 


ELL 1 P 




1 0 1 


SLIT 




257 


8 00 




3 70 


C 1 HCULAT 




3 78 
462 


A T T ACHME 




5 46 
865 


COEFF 1 C 1 




8 7 4 


HECOMB 1 N 

stuchast 


60 


28 


OXYGEN 




60 


COINCIDE 




83 
3 57 


ME MOR 


6 1 


2 


coll 1 S 1 0 




6 1 


ELECTRON 




2 85 


H 0 M 0 G E NE 

hyokodyn 


62 


62 


HELA T 1 V 1 
TRANSPOR 


63 


63 


kinetic 




73 


MAGNETO 




2 59 


NEUTRAL 




2 7 0 


STAT 1C 




278 


IONIC 




4 52 


GYRO 




469 


CO MmU N 1 C 




703 
7 86 


searing 




9 54 


CO Mp AR 1 S 


6 4 


6 4 


C 0 M P A T 1 8 
RESONA 


65 


7 4 0 
50 


COMPENS A 




65 

252 


FEED8ACK 




29 1 


THERM 1 ST 




365 


Stab 1 L 1 




7 42 


AMPL 1 F 




823 


COMPLEME 




866 

938 


COLLECT 

flip 


66 


6 6 


COMPONEN 




98 
1 3 1 


cryotron 




2 54 


T OLER A NC 




58 7 


wafer 




6 2 1 


ADHE 


6 7 


6 7 


COMPHES S 




270 


CONCENTR 




370 
4 07 


N 1 TROGEN 
KEC0M8 1 N 


68 


68 


DE NS 1 T 
OXYGEN 


69 


69 


ION 




1 09 


CONDENSE 
C A P A C 1 T 


70 


2 48 

1 2 


CONDITIO 




38 

70 


NONPER 1 0 




8B 


8 OUNDAR 




399 


CONF 1 GUR 




425 
5 6 5 


EMITTER 




6 35 


FUNCT I ON 




741 


CONIC 


7 1 


1 1 


CONFORMA 




7 1 





72 


72 


CONJUNCT 




86 






27 1 


CONST 1 TU 




3 39 


NITROGEN 




395 


ATOM 




5 \ 3 






6 06 


AD V ANT AG 




6 07 


CON VENT 1 




7 00 


FEEDBACK 




759 


RESPONSE 




8 1 8 


DISTORT 




9 3 1 


EQUAL 1 Z 




954 


STAGGER 




967 


HYBRID 






FEED 


73 


63 


GAIN 




73 






759 


BALANCED 






CONVERS 1 


7 4 


7 4 


EFF 1 C 1 EN 






CONVERT 


75 


75 






49 0 


ACTIVITY 




658 


CORRELAT 






S T A T 1 S T 1 


76 


76 






t 6 3 


COR RESPO 




22 7 






387 


CORRUGAT 




5 6 6 


D 1 ME NS 1 0 




765 


ROUGHNES 




799 


SINUSOID 






SPHEROID 


7 7 


77 


SCATTER 






SURFACE 


78 


78 


field 




1 1 1 


SHEET 




573 


WAVE 




6 1 5 


CRITICAL 


79 


79 


MAXIM 








8 0 


80 


COMPONEN 




1 1 3 


CRYOTRON 




1 59 


RESISTOR 




459 


SUPERCON 




5 0 0 


DEPOS 1 T 






UEUOR 


8 1 


8 \ 


WOUND 




4 1 6 


FILM 




755 


STOR 




7 75 


C YCLOTRO 


8 2 


82 


EXTERNAL 






1 NCOHERE 


8 3 


6 0 


resonanc 




83 


PLASMA 




5 7 9 


STATIC 




763 






95 1 


DEMODULA 






SINUSOID 


8 4 


84 


DISTORT 




3 1 6 


moduuat 




7 4 6 






9 8 5 


DEPARTUR 








85 


85 


DER 1 VAT 1 




88 


FUNCT 1 ON 




8 3 0 


REPRESEN 








86 


1 


A N 1 SOTRO 




1 2 


CERENKOV 




72 


D 1 ELEC TR 




8 6 


ELECTROS 




27 7 


1 SOTROP 




278 


PERMEA6 




395 


MEDIUM 




399 


TENSOR 




565 


MEDIA 




607 






6 1 7 


analyser 




74 1 


D 1 FFEREN 




8 1 8 




8G 





DIFFRACT 
ELEC TKOM 

solution 

80UNDAR 
I NF I N I T 
CONVEX 
SCALAR 
SCREEN 
EXPAN 
PLANE 
WEDGE 
SLIT 
WAVE 
8 00 

corruGat 

D I ME NS I 0 
PLANE 

D I RECT I 0 

D I REC T I V 
AERIAL 
ANAL Y 

D I SCHARG 

L I GH T N I N 

PRESSURE 

GASEOUS 

MERCUR 

PROSE 

SPARK 

0 I S C 0 N T I 

DISCRETE 
GAL AC T 1C 
NEBULA 
SOURCE 

D I SS I P A T 

D I STR I BU 
GEOGRAPH 
I SOTHERM 
SPATIAL 

bright 

ECCENTR I 
0R8 I T AL 
ORBIT 
REVOL 

EFFEC T I V 

CONVERS I 
EFF I C I E N 
OUTPUT 
POWER 
VOLT 

ELECTRIC 

TRANSPOR 

MOBIL 

NET 

ELEC trod 
ELECTROS 
AXES 

ACOUST I C 
AN I SO TR 0 

0 IFFRACT 
ELEC TROM 
SPHER 1 C A 
SPHERO I D 
I NF I N I T 

1 SOTROP 
MEDIUM 
SCREEN 
SPHERE 
MEDIA 
WEDGE 



86 


964 


WAVE 


1 0 1 


87 


22 


ATTACHME 


1 0 2 




4 4 


COLL 1 S 1 0 




87 


electron 






1 9 1 


N 1 TROGEN 






262 


SECONO Ar 


1 0 3 




3 1 6 


TR ANSPOR 






4 1 3 


NEUTRAL 






43 4 


Profile 






524 


D 1 FFUS 






63 4 


TARGET 






6 4 4 


V A LENC 






7 1 0 


GASES 






765 


PROBE 


1 0 4 




829 


ATOM 






979 


1 ON 


1 0 5 


88 


70 


D 1 ELECTR 






85 


ELECTROD 






88 


ELECTROS 


1 06 




225 


POTENT 1 A 






807 


THEOR 






830 


AXES 




89 


89 


ENGINEER 






4 0 4 


MACHINE 






4 3 5 


PROGRAM 






6 1 1 


C OMPUT 


1 0 7 


90 


90 


EQU 1 L 1 Br 


1 08 


9 1 


9 1 


EQU 1 vale 




92 


92 


EV APORAT 






7 43 


METAL 






866 


F 1 LM 






903 


OX i 0 






9 4 8 


V ACU 




93 


93 


exchange 






1 4 4 


interact 






53 2 


ENERGY 


1 09 


9 4 


94 


EXC 1 T AT 1 






38 1 


EXCI TED 






6 4 4 


valenc 




95 


95 


EXOSPHER 






65 1 


WH 1 ST L 






693 


EARTH 




96 


96 


EXPONENT 




97 


97 


EX TENS 1 0 


1 1 0 


98 


6 6 


C YCLOTRO 






98 


external 






332 


annular 


1 1 1 


99 


99 


EXT 1 NCT 1 






391 


1 NC 1 OEN 






43 0 


POLAR 1 Z 






4 4 4 


REFLECT 






4 52 


SCATTER 






576 


0 B L 1 QU 






577 


OR 1 ENT 






694 


ELL 1 P 




1 00 


954 


WAVE 


1 1 2 


1 0 0 


extraoro 






2 43 


f^EFLEX 1 0 






406 


magneto 






6 4 5 


valley 






8 08 


TRACE 


1 1 3 




8 76 


gyro 




1 0 1 


48 


COMPENS A 






59 


convent I 






1 0 1 


FEE08ACK 






293 


SYNCHRON 






4 6 1 


ST AB 1 L 1 




O 


499 


br I oge 




ERIC 


782 


SERVO 


1 1 4 











DR 1 V 


1 1 4 


176 


MICROPUL 


LOOP 




37 1 


D 1 STURB 






379 


equator 


FERROELE 




529 


0 YN AMO 


CAP AC 1 T 




59 1 


PULS AT 






7 1 2 


GIANT 


FERROM AG 




733 


LOCAL 


MAGNET 1 C 




762 


POLAR 


M A G N E ,1 S 




797 


SOUTH 


nonmagne 




803 


STORM 


MAGNET 1 




8 9 4 


MAIN 


THEOR 




965 


BAY 


SPIN 










1 1 5 


1 1 5 


GEOPH YS 1 


FILAMENT 




1 4 8 


1 NTERNAT 


CURRENT 




85 7 


EAST 






960 


YEAR 


FLUCTU AT 








SHOT 


1 1 6 


1 1 6 


GERM AN 1 U 






1 6 0 


JUNCTION 


ATTACHME 




5 9 9 


RECT 1 F 


FORMAT 1 0 




79 2 


S 1 L 1 C 


1 ON 1 Z AT 1 








R ECOMB 1 N 


1 1 7 


1 1 7 


gradient 


OXYGEN 




159 


I SOTHERM 


ALT 1 




326 


VERTICAL 


SOFT 




529 


DYNAMO 


FRACT 1 ON 


1 1 8 


1 1 8 


HAMILTON 


B AND W 1 D T 


1 1 9 


1 1 9 


HARMONIC 


frequenc 








SELECT 1 V 


1 2 0 


1 1 3 


GEOGRAPH 


TUNABLE 




1 2 0 


HEM 1 SPHE 


ADJUST 




1 6 2 


L AT 1 TUOE 


NARROW 




3 2 4 


V AR 1 AT 1 0 


R ESO N A 




463 


STATION 


SWEEP 




752 


NORTH 


LOCK 




797 


SOUTH 


TONE 








1 OL 


1 2 1 


1 2 


AN ( SOTRO 


M 1 X 




4 4 


COLL 1 S 1 0 






1 2 1 


HOMOGENE 


CONFORMA 




4 0 1 


KINETIC 


DER 1 VAT 1 




6 2 1 


STATIC 


FUNC T 1 ON 




635 


TENSOR 


MATHEM AT 




7 03 


FIELD 


NUMER 1 CA 




707 


FLUID 


POLYNOM 1 




96 7 


BOD 


TRANSFER 








TRANS 1 EN 


1 22 


1 2 2 


HOR 1 ZONT 


EVALUAT 




1 85 


movement 


POLE 




3 2 5 


VERTICAL 


REAL 




956 


WIND 


FUNO AMEN 


1 23 


4 4 


COLL 1 S 1 0 


FOURIER 




1 23 


H YDROO YN 


0 AMP 1 




387 


GASEOUS 






406 


MAGNETO 


0 1 SCRETE 




5 8 7 


PLASMA 


GALACT 1 C 




725 


IONIC 


R AO 1 A T 1 0 




726 


1 ON 1 Z 


NE8ULA 








SOURCE 


1 2 4 


124 


HYDROGEN 


GALAX 




765 


PROBE 


R AO 1 0 








flux 


1 25 


1 2 5 


HYDROMAG 


ARC 




5 9 1 


PU LS AT 






7 07 


FLUID 


APPROX 1 M 








asymmetr 


1 26 


1 26 


H YPERBOL 


gauss 1 AN 








M 1 X 


1 27 


1 2 7 


HYPOTHES 


D 1 ST R 1 BU 


1 28 


4 


AOM 1 TTAN 


GEOGRAPH 




1 28 


1 MPEDANC 


HEM t SPHE 




1 3 8 


1 NSERT 1 0 


LATITUDE 




233 


OU ADR 1 PO 


STATION 




298 


TERMINAL 


DAILY 




4 8 7 


ACTIVE 


SOUTH 




4 9 8 


BRANCH 


WORLD 




502 


CASC AO 






556 


LADDER 


geomagne 




7 1 9 


IMAGE 



852 

8 92 

1 02 

3 4 1 

1 03 

1 66 

1 6 7 

1 93 

4 05 

807 

934 

1 04 

36 1 

1 05 

9 27 

22 

1 06 

1 53 

2 40 

580 

8 26 

933 

1 07 

29 

1 08 

263 

4 78 

488 

5 72 

6 0 1 

8 06 

890 

943 

9 78 

984 

56 

69 

1 09 

1 72 

1 9 9 

22 1 

3 1 1 

3 1 3 

380 

9 1 1 

920 

1 1 0 

386 

680 

78 

1 1 1 

235 

573 

6 1 5 

709 

77 1 

67 1 

96 4 

\ 5 

1 9 

} 1 2 

9 8 4 

80 

1 1 3 

1 20 

1 62 

463 

6 79 

797 

822 

1 1 4 

87 



88 





1 28 


738 


MATCH 


1 45 




1 29 


1 29 


1 MP8RFEC 








469 


SURFACE 








b y 9 


EXACT 






1 30 


1 3 0 


Impurity 








fa 4 4 


V ALENC 






1 3 1 


66 


CYCLOTRO 








1 3 1 


1 NCOHERE 








2 4 6 


RELAT 1 V 1 








4 1 3 


neutral 


1 46 






4 52 


SCATTER 








62 1 


STATIC 


1 47 




1 32 


1 32 


1 N C R E M E N 






1 33 


1 33 


1 NDEPE NO 


1 48 




1 3 4 


1 3 4 


1 NDUCT AN 








1 3 5 


1 NDUCTOR 








3 4 1 


C APAC 1 T 


1 49 






3 45 


circuit 








393 


1 NUUCT 1 








823 


WOUNO 








84 1 


COIL 






1 35 


1 3 4 


1 NOUCT an 








1 3 5 


inductor 








3 4 1 


C APAC 1 T 








4 3 9 


REACT 1 V 






1 36 


1 36 


1 nfurmat 


1 5 0 






4 4 1 


REAO 1 NG 








4 8 4 


ACCESS 








51 1 


COMP U T 








63 2 


system 








fa 8 7 


0 1 G 1 T 


1 5 1 






742 


ME MOR 








800 


SPEEO 


1 52 






93 8 


STOR 








9 4 0 


TAPE 






1 37 


1 3 7 


1 NHOMOGE 


1 53 




1 38 


1 28 


1 MPEO ANC 








1 38 


1 NSERT 1 0 








233 


U U AOR 1 PO 








29 9 


TERM 1 NAt 








49 8 


BRANCH 








556 


L AOOER 








7 1 9 


Image 








83 1 


8 ANO 


1 5 4 






867 


F 1 LT 








893 


LOSS 








9 05 


PASS 






1 3 9 


1 39 


1 NSTAB I L 






1 4 0 


1 4 0 


1 nstrume 








6 05 


ROCKET 


1 55 




1 4 1 


2 0 


ASYMPTOT 








1 4 1 


INTEGRAL 








27 1 


SOL UT 1 ON 








696 


EQUAT 








700 


EXPAN 




V 




796 


SOLVE 


1 56 






8 1 8 


WEOGE 






1 42 


1 42 


1 NTEGR AT 






1 4 3 


1 4 3 


1 N T E N S 1 T 








1 49 


1 NTERPL A 








3 00 


terrestr 








5 1 5 


COSMIC 






1 4 4 


93 


exchange 








1 4 4 


INTERACT 








1 67 


MAGNET 1 S 




O 

ERIC 




382 


EXC 1 TON 






532 
5 69 


ENERGY 

MOMENT 

















1 56 


679 


DAILY 


1 NTERFER 




752 


NORTH 


RAO 1 AT 1 0 




797 


SOUTH 


TELESCOP 








W A VELENG 


1 57 


1 57 


1 R R E G UL A 


aerial 




243 


REFLEX 1 0 


0 1 AMET 




3 1 9 


TURBULEN 


SOURCE 




49 4 


ASPECT 


ASTRO 




537 


FADING 


R AO 1 0 




6 1 9 


SPREAD 


SUN 




655 


ALIGN 






858 


ECHO 


1 NTERMED 










1 58 


1 58 


1 R E V E R S 


1 NTERNAL 




303 


THERMOD Y 


EMI TTER 




832 


BASE 


FEEO 




901 


NOTE 


GEOPH YS 1 


1 59 


2 1 


ATMOSPHE 


1 NTERNAT 




22 


attachme 


YEAR 




80 


0 1 STR 1 BU 






1 1 7 


gradient 


ARGUMENT 




1 59 


ISOTHERM 


1 NTENS 1 T 




296 


TEMPER AT 


1 NTERPL A 




390 


HEATING 


TERRESTR 




780 


SCALE 


COR P U SC 








COSMIC 


1 6 0 


1 1 6 


GERM AN 1 U 


EARTH 




1 60 


JUNCTION 


SPACE 




3 1 4 


TRANS 1 ST 


AREA 




5 4 6 


H YBR 1 0 


RAY 




65 6 


ALLOY 






792 


S 1 L 1 C 


1 NTER VAL 








R ECORO 


1 6 1 


1 6 1 


L ABORATO 


TRAIN 








TIME 


1 62 


1 1 3 


GEOGRAPH 






1 2 0 


HEM 1 SPHE 


1 NV AR 1 AN 




1 62 


L at 1 TUOE 






1 8 4 


MORPHOLO 


1 NVERS 1 0 




379 


EQUATOR 


POP UL AT 1 




679 


DAILY 


SPL 1 




762 


polar 






768 


QUIET 


FORMAT 1 0 




822 


WORLD 


1 ON 1 Z AT 1 




96 0 


YEAR 


1 ONOGRAM 








1 ONOSONO 


1 63 


2 1 


ATMOSPHE 


MOVEMENT 




76 


0 1 SCHARG 


SPOR AO 1 C 




1 63 


L 1 GHTN 1 N 


EQUATOR 




4 73 


THUNOER 


0 1 FFUS 




6 03 


RETURN 






626 


STROKE 


1 ON 1 Z AT 1 




672 


CLOUO 


1 ONOGRAM 




7 06 


FLASH 


1 ONOSONO 








NONSEASO 


1 64 


164 


LONG 1 TUO 


SPOR AO 1 C 




474 


TORSION 


VERTICAL 




58 7 


PLASMA 


VALLEY 










1 65 


1 65 


L UM 1 NESC 


1 ON 1 Z AT 1 




375 


electro 


1 ONOGRAM 








1 ONOSONO 


1 66 


1 03 


FERROM AG 


SOUNO 1 NG 




1 66 


MAGNETIC 


R ECORO 




1 6 7 


magnet I S 


SPR E AO 




385 


FERRITE 






405 


MAGNET 1 


1 ONOSPHE 




45 1 


S ATURAT 


MOVEMENT 




569 


moment 


reflex I 0 




707 


FLUID 


SOUNDING 




708 


FORCE 


VER T 1 CAL 




7 1 1 


GAUSS 


DAYTIME 




854 


DRUM 


0 1 UR N AL 




877 


HEAD 


EQUATOR 








STATION 


1 67 


1 03 


F ER ROM AG 


V 1 RTUAL 




1 44 


1 NTER ACT 


ARCTIC 




1 66 


magnet I C 


DYNAMO 




167 


magnet 1 S 


SEASON 




569 


moment 


SPREAD 




934 


SPIN 


SUMMER 








WINTER 


} 68 


1 68 


magnetom 



1 45 

2 35 

2 95 

3 28 

490 

523 

6 1 5 

6 6 1 

7 7 1 

9 93 

1 46 

1 47 

3 77 

865 

1 1 5 

1 48 

960 

1 6 

1 43 

1 49 

3 00 

3 58 

5 1 5 

6 9 3 

7 98 

8 28 

9 87 

I 50 

597 

8 1 1 

942 

1 5 1 

1 52 

2 22 

8 0 I 

1 06 

1 53 

1 54 

1 55 

1 85 

2 80 

37 9 

524 

1 53 

1 54 

1 55 

) 96 

280 

3 25 

6 45 

1 53 

1 54 

1 55 

272 

597 

6 1 9 

1 56 

t 85 

2 4 3 

272 

325 

362 

372 

379 

463 

48 1 

4 93 

529 

6 09 

6 1 9 

627 

6 52 



89 



1 6 8 


563 


ME ASUR 


1 85 


1 69 


1 69 


M AGNETOR 






264 


SEW 1 CO ND 






35 0 


C 0 M D U C T 


1 8 6 




7 03 


field 


1 8 7 


1 7 0 


1 70 


MAGNETOS 






474 


TORSION 


1 88 


1 7 1 


1 7 1 


M A Cl N 1 T U D 


189 


1 72 


1 0 


ANALOGUE 






1 09 


F U M C T 1 ON 






1 72 


MATHEMAT 






2 1 4 


photocel 






248 


Re PRE SEN 


1 9 0 




538 


FINITE 






62 2 


steady 






63 1 


SYMBOL 






647 


VECTOR 






696 


E Q U A T 




1 7 3 


1 73 


MECHAN 1 C 


1 9 1 


1 74 


1 74 


MEGHAN 1 S 




1 75 


1 75 


M 1 CROM 1 N 






1 B 1 


M 1 N 1 ATUR 






345 


circuit 






52 0 


DESIGN 






735 


LOGIC 






748 


modul 




1 7 6 


1 1 4 


GeOMAGNE 






1 76 


MICROPUL 


1 9 2 




59 1 


PULSAT 




1 7 7 


1 77 


M 1 CROWAV 


1 93 




276 


spectros 




1 78 


6 


afternoo 






1 78 


MIDNIGHT 


1 94 




1 96 


NONSEASO 






582 


PERIOD 






733 


LOCAL 


1 95 




74 0 


MAXIM 






965 


BAY 




1 7 9 


1 79 


M 1 LL 1 MET 


1 96 




289 


S U B M IL L 1 






38B 


G E N E R A T 






9 54 


WAVE 




1 8 0 


1 8 0 


M 1 LL 1 M 1 C 






388 


GENER AT 






766 


Pulse 






922 


R 1 SE 




1 8 1 


1 75 


M 1 CR OM 1 N 






1 8 1 


M 1 N 1 ATUR 




1 82 


1 82 


momentum 


1 9 7 


1 83 


532 


Energy 


1 9 8 


32 


bistable 




33 


blocking 






1 83 


monostab 






1 9 0 


M U L T 1 V 1 B 






3 1 4 


TRANS 1 ST 


1 99 




345 


CIRCUIT 




477 


TR 1 GGER 






6 1 0 


SELECT 


2 0 0 




835 


Bias 




869 


flip 




1 84 


1 62 


L AT 1 TUDE 






1 84 


MORPHOLO 






371 


disturb 






489 


ad VANC 






803 


STORM 






f>9 4 


T 1 D 




FRir' 


1 22 


HOR 1 20NT 





1 ON 1 Z AT 1 
1 ONOSPHE 


2 0 1 


5 


ADVANTAG 


MOVEMENT 




32 


bistable 


MULT 1 L AY 




201 

29 2 


DPER AT 1 0 

superreg 


MULT 1 PL 1 




663 

735 


BASIC 

LOGIC 


MULT 1 POL 




869 

962 


FLIP 

ADD 


M ATR 1 


2 02 


2 02 


ORTHOGON 


MULT 1 ST A 
RESPONSE 


2 03 


3 3 


BLOCKING 


OVERSHO 




2 03 


0SC1 LLAT 


STAGGER 




38 1 


EXCITED 


B 1 stable 




4 1 0 
6 0 1 


NATURAL 
R E S 0 N A 


BLOCKING 




68 0 


D AMP 1 


MONOSTAB 




89 0 


LOCK 


MULT 1 V 1 B 




926 


SELF 


CIRCUIT 

TRIGGER 


2 04 


2 04 


0 SC 1 LL OG 


flip 




535 


EXTREM 


CONCENTR 


2 05 


1 8 


ASSOC 1 AT 


CONST 1 TU 




2 05 


OUTBURST 


ELECTRON 




667 


BURST 


nitrogen 




698 


ERUPT 


REACTION 




7 0b 


FLARE 


ATOM 1 C 




794 


SOLAR 


HELIUM 




0 03 


STORM 


OXYGEN 

GASES 


206 


206 


PARALLEL 


1 ON 


207 


207 


P AR AMAGN 


N 0 N L 1 N £ A 




254 


RESONANC 


ROW 


2 08 


2 08 


PARTICLE 


FERROM AG 




246 


RELAX 1 V 1 


NONM AGNE 




3 09 


TR A JECTO 


SPIN 




5 7 1 


MOVING 


CONDITIO 




5 9 0 
669 


PROTON 

CHARG 


NONPER ! 0 




7 00 


FORCE 


N 0 N R E C 1 P 




757 
B3 4 


OUTER 

BELT 


ELEMENT 




944 


TRAP 


PASS 1 V 


209 


209 


PENE TR AT 


AFTERNOO 




6 8b 


DEPTH 


1 ONOGR AM 
MIDNIGHT 


2 1 0 


2 1 0 


PER FORMA 


NONSEASO 
VAR 1 AT I 0 


2 1 1 


2 1 1 


PERM ANEN 


SUNRISE 
ANOM AL 


2 1 2 


1 


ACOUST 1 C 


dens 1 T 




2 1 2 


PERPEND 1 


HEIGHT 




647 


VECTOR 


S E A f , 0 N 




759 


PLANE 


daily 




786 


SHEET 


LAYER 


2 1 3 


2 1 3 


PHE NOMEN 


NONUN 1 FO 




8B3 


JUMP 


NORMAL 1 Z 


21 4 


1 72 


MATHEMAT 


TERM 1 NAT 




2 1 4 


PHOTOCEL 


TA8UL A 




3 02 


THERMOCO 


F 1 LT 


21 5 


2 1 5 


PHOTOCON 


F U N C T 1 ON 




375 


ELECTRO 


N UMER 1 CA 


2 1 6 


34 


BREMSSTR 


OBSER VAT 




2 1 6 


PHOTOELE 


telescop 




743 


METAL 


TERRESTR 




9 09 


PHOT 


DETON AT 
JUP 1 TER 


21 7 


21 7 


PHOTOGR A 


B R 1 Gh T 




596 


RECE 1 V 


NE8UL A 




597 


RECORD 


VISUAL 




649 


VISUAL 


ASTRO 

MAJOR 


2 1 8 


21 8 


PHOTOSEN 


QUIET 




456 


SH AP 1 NG 


SOUTH 




675 


COUNT 



1 53 

1 56 

1 85 

1 8 6 

1 87 

1 88 

739 

1 89 

■257 

4 t 7 

462 

32 

33 

1 83 

1 90 

3 45 

477 

8 69 

52 

58 

87 

1 9 1 

238 

495 

545 

580 

7 1 0 

9^ B 

1 9T 

990 

1 03 

1 93 

934 

54 

1 94 

1 95 

3 76 

581 

6 

1 54 

1 78 

1 96 

324 

466 

492 

5 1 9 

544 

6 09 

679 

72B 

1 9 7 

1 98 

299 

633 

867 

1 09 

1 99 

2 00 

295 

3 00 

3 6 6 

400 

5 00 

573 

649 

66 1 

736 

768 

797 



90 





2 1 8 


7 53 


OPTIC 


236 




2 1 S 


2 1 9 


P 1 EZOELE 








36 0 


CRYST A L 






2 2 0 


2 2 0 


POLARITY 


237 




22 1 


1 09 


FUNCT 1 ON 








22 1 


POLYNOM 1 








920 


Real 


238 




222 


1 52 


1 NVER5 1 0 








222 


POPULA T 1 








73 7 


MASER 








9 1 5 


Pump 


23 9 




223 


223 


POSITION 








5 04 


centre 






22 4 


22 4 


POSITIVE 








4 1 1 


NEGAT 1 V 


240 




225 


88 


ELECTR OS 








22 5 


POTENT 1 A 








336 


BARRIER 






226 


226 


POTENT 1 0 








25 0 


Res 1 ST An 






227 


76 


D 1 SCH ARG 


24 1 






227 


PRESSURE 








387 


gaseous 


2 42 






424 


PERIGEE 








545 


helium 








7 1 0 


gases 








765 


Probe 


243 




228 


228 


PR 1 NC 1 P A 






229 


229 


P R OB A B 1 L 


2 44 




2 3 0 


23 0 


PROGRESS 






23 1 


23 1 


propagat 








406 


magneto 


2 45 






74 1 


MEO 1 A 








856 


DUCT 


2 46 






9 54 


WAVE 






232 


232 


PR OP OR T 1 








398 


1 NVERSE • 






233 


1 2 8 


1 MPEDANC 








1 38 


1 NSER T 1 0 








23 3 


GU ADR 1 P 0 








298 


terminal 








3 4 0 


CANONIC 








4 1 2 


Network 


247 






498 


BRANCH 








502 


C ASC AD 








58 1 


P A S S 1 V 


2 48 






7 1 9 


IMAGE 






234 


234 


QUADRUPO 








4 1 5 


NUCLEAR 








737 


MASER 






23 5 


3 8 
1 1 1 


CERENKOV 

GALACTIC 


2 49 






1 45 


I NTERF ER 


250 






23 5 


R AD 1 AT 1 0 






328 


WAVELENG 








363 


D EC 1 M E T 


25 1 






400 


JUPITER 






500 


Bright 




, 




5 1 4 


CORONA 








5 73 


nebula 




i 




7 09 


Galax 


252 


i 




834 


belt 






87 1 


flux 








988 


Red 








9 93 


SUN 




ERIC 


236 


2 1 


ATMOSPHe 















R AD 1 OSON 








refract 


253 


253 


RESOLUT 1 


MEASUR 




49 0 


AERIAL 


INDEX 




600 


RESOLV 


DATA 




879 


HIGH 






925 


SCAN 


reactanc 








RE ACT 1 V 


254 


68 


CYCLOTRO 


ROW 




207 


P AR AMAGN 






24 7 


RELAXAT 1 


N 1 TROGEN 




254 


RESONANC 


REACTION 




288 


S TOCH AST 


OXYGEN 




4 1 5 


NUCLEAR 






702 


F ERR 1 


REC 1 PROC 








TERMINAL 


255 


255 


RESONATO 


PASS 1 V 




4 1 0 


NATURAL 


EQUAL 




668 


C A V 1 T 


PAIR 










2 56 


1 9 


AS YMMETR 


ATTACH ME 




59 


C 0 N V E 14 T 1 


COEFF 1 C 1 




1 8 9 


MULTI STA 


CONCENTR 




256 


RESP0f4SE 


FORMATI 0 




263 


S ELECT 1 V 


RECOMB 1 N 




3 1 3 


TRANS 1 EN 


ATOMIC 




3 7 8 


EQUAL 1 Z 


OXYGEN 




4 1 7 


0 VER SHO 


FALL 




454 


SEGMENT 






462 


STAGGER 


REFERENC 




47 1 


TELEV 1 S 






782 


SERVO 


reflecto 




883 


JUMP 


TELESCOP 




9 1 0 


PLOT 


P AR ABOL 




93 7 


STEP 


extraord 


257 


25 7 


RESTR 1 CT 


1 ONOSPHE 




4 1 7 


0 VERSHO 


1 RR EGUL A 




9 9 0 


ROW 


REFLEX 1 0 










2 58 


258 


REVERS 1 B 


REGENERA 




6 6 5 


8 1 N AR 


S I'NCHRON 




6 7 5 


COUNT 


D 1 V 1 










2 59 


63 


CORRU(iAT 


RELATIVE 




259 


ROUGH NES 






444 


REFLECT 


COLL 1 S 1 0 




469 


SURFACE 


1 NCOHERE 




563 


MEASUR 


particle 








RELAT 1 V 1 


260 


1 7 


AR T 1 F 1 C 1 


KINETIC 




26 0 


S ATELL 1 T 


GROWTH 




366 


DETON AT 


INJECT 




384 


FARADAY 


MOVING 




4 1 6 


ORB 1 TAL 


CH ARG 




42 4 


PERIGEE 


FORCE 




46 0 


SPUTNIK 






55 7 


LAUNCH 


REL AXAT i 




69 3 


EARTH 


RESONANC 




755 


OR 8 1 T 






8 1 5 


V EH 1 C 


ADM 1 TTAN 




85 0 


DRAG 


D E R 1 V A T 1 








MATHEM AT 


26 1 


26 1 


SC 1 NT 1 LL 


REPRESEN 




93 6 


STAR 


COMPLEX 








M ATR 1 


262 


87 


ELECTRON 






262 


S ECONDAR 


RES 1 DUAL 




33 8 


8 0 M8 ARD 






43 2 


PRIMARY 


POTENT 1 0 




53 1 


EM 1 SS 1 


RES \ STAN 




53 2 


ENERGY 






63 4 


TARGET 


R ES 1 ST 1 V 




743 


METAL 


RESISTOR 




903 


0 X 1 D 


CAP AC 1 T 








REACT 1 V 


263 


1 08 


FREQUENC 






257 


RESPONSE 


CRYOTRON 




263 


SELECT 1 V 


RESI STI V 




378 


EQUAL 1 Z 


RESISTOR 




462 


STAGGER 


protect 




499 


8 R I DGE 


CARBON 




520 


DESIGN 


WOUND 




572 


NARROW 


F 1 X 




905 


PASS 



2 3 6 

4 45 

5 6 3 

7 2 1 

848 

2 3 7 

439 

9 9 0 

1 9 1 

2 38 

5 8 0 

2 3 9 

2 9 8 

5 8 1 

6 95 

9 04 

2 2 

42 

52 

1 06 

2 4 0 

495 

5 8 0 

8 6 3 

2 4 1 

242 

295 

4 1 8 

1 00 

1 56 

1 5 7 

2 4 3 

2 4 4 

2 9 3 

849 

245 

44 

1 3 1 

2 08 

2 4 6 

4 0 1 

542 

55 1 

5 7 1 

669 

7 08 

247 

2 54 

4 

69 

1 72 

2 4 8 

3 49 

739 

249 

2 26 

250 

2 5 1 

2 52 

3 4 1 

439 

65 

2 5 1 

252 

43 6 

501 

8 23 

973 



26 4 


1 6 9 


MAGNETOR 






26 4 


SEM 1 COND 






3 42 


CARRIER 




265 


2 6 5 


SEM 1 0 1 UR 






3 2 4 


VAR 1 AT 1 0 


28 1 




529 


DYNAMO 






9 0 8 


PH AS 






9 8 3 


LUN 






994 


T 1 D 


28 2 


266 


266 


SENS 1 T 1 V 






49 4 


ASPECT 


2 8 3 


267 


267 


SEQUENCE 




26 8 


2 6 8 


SEQUENT 1 


284 


269 


269 


S 1 MULT AN 






496 


AURORA 






858 


ECHO 




2 7 0 


6 3 


CORRUGAt 


285 




6 7 


DEMODULA 






2 7 0 


SINUSOID 






3 7 0 


DISTORT 




27 1 


2 0 


ASYMPTOT 






72 


diffract 






1 4 1 


Integral 


286 




2 7 1 


solution 






2 78 


SPHERO ID 






3 3 9 


8 OUnDAR 






3 9 5 


1 NF 1 N 1 T 






53 8 


Finite 






6 06 


SCALAR 






fa 0 7 


SCREEN 






6 4 3 


UNIQUE 


287 




696 


EQUAt 






69 9 


EXACT 






7 0 0 


EXPAN 


288 




796 


SOLVE 




272 


1 55 


1 ONOSOND 


2 89 




1 56 


1 ONOSPHE 






2 7 2 


SOUND 1 NG 






3 2 5 


vertical 






48 1 


VIRTUAL 






858 


ECHO 


2 9 0 


27 3 


273 


SPECIMEN 




27 4 


2 7 4 


SPECTROG 


29 1 




2 76 


SPECTROS 




275 


2 7 5 


SPECT ROM 






2 76 


SPECTROS 


292 


276 


1 7 7 


M 1 CR 0 W A V 






2 7 4 


SpECTROG 






2 7 5 


SPECTROM 






2 76 


SPECTROS 




277 


8 6 


electrom 


29 3 




2 7 7 


SPH ER 1 C A 






3 5 0 


Conduct 






5 1 6 


C YL 1 ND 






7 59 


Plane 




278 


6 3 


CORRUGAt 






8 6 


electrom 






27 1 


SOLUT 1 ON 


294 




2 7 8 


SPHEROID 




6 0 6 


SCALAR 






7 0 0 


EXPAN 




279 


967 


BOD 


295 


279 


spontane 




5 0 6 


CDHERE 






53 1 


EM 1 SS 1 




O a Q 

ERIC 


1 53 
1 54 


1 ON 1 ZAT 1 
1 ONOGR am 













SPORADIC 


295 


596 


RECE 1 V 


VERTICAL 




6 6 1 


astro 


HEIGHT 




7 7 1 


RADIO 


RECORD 




925 


SCAN 


ECHO 










296 


1 59 


ISOTHERM 


SPUR 1 OUS 




296 


tempekat 


DESIGN 




304 


thermoel 


SELECT 




329 


AMBIENT 


MODE 




924 


ROOM 


STANDARD 


29 7 


297 


TEMPORAL 






372 


DIURNAL 


STANDING 




6 09 


SEASON 


EXCITED 








PLASMA 


29 8 


4 


ADMI TTAN 


WAVE 




1 2 8 


1 MPEDA NC 






2 3 3 


QU ADR 1 PO 


ST AT 1 ON A 




239 


REC 1 PROC 


finite 




298 


TERMINAL 


GROWTH 




4 1 2 


NETWORK 


EQUAT 




58 1 


P ASS 1 V 


ACTIVITY 


299 


1 3 B 


1 NSERT 1 0 


CORRELAT 




1 98 


NORM AL 1 Z 


S T A T 1 ST 1 




299 


T E R M 1 N A T 


SUNSPOT 




4 39 


RE AC T 1 V 


PERIOD 




49 8 


BRANCH 


STORM 




556 


LAUDER 






738 


MATCH 


COEFF 1 C 1 




9 05 


PASS 


RESON ANC 








STOCHAST 


300 


1 43 


1 NTENS 1 T 


RANDOM 




1 49 


I NTERPL A 


SPECTR 




2 00 


OBSERVAT 


THEOR 




3 00 


TERRESTR 


SPIN 




5 1 5 


COSMIC 






693 


EARTH 


STRENGTH 




8 0 3 


STORM 


FIELD 




987 


KAY 






993 


SUN 


STRUCTUR 








FINE 


3 0 1 


48 


C 0 M P E N S A 






301 


T HERM 1 ST 


M 1 LL 1 MET 




329 


AMB 1 ENT 


SUBM 1 LL 1 




46 1 


STAB 1 L 1 


GENER AT 








SEAM 


302 


2 1 4 


PHOTOCEL 


WAVE 




3 02 


THERMOCO 


SUCCESS 1 


303 


1 58 


1 RREVERS 


STROKE 




303 


THERMOD Y 


CRYOTRON 


30 4 


296 


temper AT 


SUPERCON 




304 


THERMOEL 


DEPOSIT 










305 


305 


T H 1 CKNES 


OPER AT 1 0 




785 


SHEAR 


SUPERREG 








PAR AMET 


3 0 6 


3 06 


threshql 


AMPL 1 F 








DIODE 


3 0 7 


307 


THYRATRO 


NO 1 S 




345 


CIRCUIT 






4 4 6 


REGUL AT 


feedback 




450 


S ATUR A8 


REGENERA 




468 


SUPP L 1 £ 


SYNCH RON 




579 


OUTPUT 


DISTORT 




951 


VOLT 


LEADING 








INPUT 


308 


50 


COMPONE N 


TONE 




308 


TOLER ANC 






52 0 


DESIGN 


SYNTHES 1 




950 


VIEW 


CANONIC 








ELEMENT 


309 


208 


PARTICLE 


NETWORK 




3 09 


T R A JECT 0 






669 


CH ARG 


1 NTERFER 








OBSERVAT 


3 1 0 


3 1 0 


TR ANSDUC 


reflecto 








telescop 


3 1 1 


1 09 


FUNC T 1 0 N 


AERIAL 




3 1 1 


TRANSFER 


D 1 AMET 




41 2 


NETWORK 


MIRROR 




9 1 1 


POLE 



2 80 

3 25 

5 4 4 

59 7 

658 

2 8 1 

520 

6 1 0 

89 8 

2 82 

2 8 3 

38 1 

5 87 

9 54 

2 8 4 

538 

542 

696 

2 

61 

2 85 

467 

582 

8 03 

42 

2 54 

2 86 

^95 

6 1 6 

8 07 

9 3 4 

287 

7 03 

2 88 

868 

1 79 

2 89 

3 88 

8 3 3 

9 54 

290 

6 26 

65 

2 9 1 

3 65 

2 0 1 

292 

4 1 9 

49 1 

6 89 

9 00 

1 01 

2 4 4 

2 9 3 

3 70 

403 

724 

9 43 

294 

340 

3 76 

4 1 2 

1 45 

2 00 

2 42 

295 

490 

523 

568 



325 



3 1 2 


56 


CONFORM A 






3 1 2 


TRANSFOR 






3 4 0 


CANONIC 






386 


FOURIER 






80 1 


SPLIT 


326 


3 1 3 


1 09 


FUNCT 1 ON 






257 


response 






3 1 3 


TRANS 1 EN 






4 1 7 


OVERSHO 






5 5 9 


LINEAR 


327 




72 4 


INPUT 






937 


Step 


328 


3 1 4 


3 2 


bistable 






1 60 


JUNCTION 






1 83 


MO NO S T A B 






3 1 4 


TRANS 1 ST 






3 45 


CIRCUIT 






3 48 


COLLECT 


329 




3 7 7 


EMITTER 






47 7 


TR 1 GGER 






6 6 3 


basic 






67 1 


class 






8 3 5 


B i AS 






838 


CHOP 


3 30 




869 


Flip 






9 3 5 


stab 


3 3 1 


3 1 5 


3 1 5 


T R A N S 1 T 1 


3 3 2 


3 1 6 


4 4 


COLL 1 S 1 0 






8 4 


ELECTRIC 






87 


electron 






3 1 6 


TRANSPOR 


3 3 3 




40 1 


KINETIC 






62 1 


STATIC 






70 3 


F 1 eld 






726 


1 ON 1 Z 


334 




7 46 


MOBIL 






976 


GAS 


335 


3 1 7 


3 1 7 


TR ANSVER 






62 1 


STATIC 






V 0 3 


Field 


3 36 


3 1 8 


3 1 8 


TR AvELL 1 






95 4 


Wave 




3 1 9 


1 57 


1 RREGUL A 


3 37 




3 1 9 


TURBULEN 






56 7 


meteor 






78 0 


SCALE 






8 1 0 


TRAIL 


338 


320 


320 


UN 1 D 1 R EC 




32 1 


32 1 


UN 1 VERSA 


3 3 9 


322 


322 


unstable 




323 


3 2 3 


VAR 1 ABLE 






973 


F 1 X 




324 


1 2 0 


HEM| sphe 






1 96 


NONSE ASO 






265 


SeM 1 D 1 UR 


3 4 0 




324 


V A R 1 A T 1 0 






3 72 


DIURNAL 






6 09 


SEASON 






62 7 


summer 






6 52 


Winter 






6 79 


daily 






882 


Hour 






960 


Yeah 


3 4 1 




99 4 


T 1 D 


325 


1 1 7 


Gradient 






1 22 


HOR 1 ZONT 






1 54 


1 ONOGR AM 






1 56 


1 ONOSPH E 




o 


2 72 


SOUNDING 




ERIC 


280 


Sporadic 













VERTICAL 








HEIGHT 


3 4 2 


26 4 


SEM 1 COND 


LAYER 




3 42 


CARRIER 


SCALE 








LUN 


3 43 


3 4 3 


central 






42 1 


PASSAGE 


BLOCKING 




46 7 


SUNSPOT 


WAVEFORM 




582 


PERIOD 


SELECT 








PULSE 


3 4 4 


3 4 4 


CHANNEL 


W AVEGU 1 D 


3 4 5 


32 


BISTABLE 






1 3 4 


1 NDUCTAN 


CENT 1 MET 




175 


M 1 CROM t N 


1 NTERFER 




1 8 3 


MONOSTAB 


RAD 1 AT 1 0 




1 9 0 


MULT 1 V 1 B 


WAVELENG 




307 


TH YR ATRO 


DEC 1 MET 




3 1 4 


TR ANS 1 S T 


METRE 




3 4 5 


CIRCUIT 






4 4 0 


REACTOR 


TEMPER AT 




456 


SHAPING 


THERM 1 ST 




47 7 


TRIGGER 


AMBIENT 




497 


B ALANC 


LEADING 




852 


DR 1 V 


COOL 










3 4 6 


3 46 


CLASS 1 C 


ANALOGY 




6 3 5 


TENSOR 






703 


FIELD 


ANGULAR 




7 96 


SOLVE 


RADIO 




807 


THEOR 






8 3 0 


AXES 


EXTERNAL 








ANNULAR 


3 47 


3 47 


CL ASS 1 F 


E C L 1 P S 










3 4 8 


49 


COMP LEME 


ANTENNA 




3 1 4 


TRANS 1 ST 


AERIAL 




3 4 8 


COLLECT 


RECE 1 V 




377 


EMITTER 


AVERAGE 


3 4 9 


248 


REPRESEN 






3 4 9 


COMPLEX 


BALLOON 








COSMIC 


3 5 0 


1 


ACOUST 1 C 


ALT 1 




1 6 9 


MAGNETOR 


RAY 




277 


SPHER 1 C A 






3 50 


CONDUCT 


POTENT 1 A 




5 1 3 


CONVEX 


BARRIER 




5 1 6 


C YL 1 ND 


SURFACE 




6 07 


SCREEN 






6 1 7 


SPHERE 


COMMUN 1 C 




6 4 4 


V ALENC 


BEARING 




6 99 


EXACT 


FARADAY 




707 


FLUID 






7 8 6 


SHEET 


SECOND AR 




930 


SKIN 


BOMBARD 








EM 1 SS 1 


35 1 


35 1 


CO NSER V 






532 


ENERGY 


CONDITIO 








DIFFRACT 


352 


352 


CONST AN 


SOLUT 1 ON 








BOUNDAR 


353 


353 


CONTACT 


FINITE 




76 1 


POINT 


UNIQUE 








CONIC 


35 4 


3 5 4 


CONT 1 NU 


PLANE 










355 


355 


CONTOUR 


Q U A D R 1 P 0 








SYNTHES 1 


356 


3 0 


8 1 D 1 R EC T 


TRANSFOR 




3 56 


CONTROL 


CANONIC 




455 


SENSING 


NETWORK 




5 8 8 


PRECIS 


P A SS 1 V 




58 9 


PRESET 


MATR 1 




760 


MOTOR 


CONDENSE 


3 57 


6 0 


CON VERS 1 


F E R R 0 F L E 




357 


convert 


1 NDUCTAN 




4 03 


LEADING 


1 NDUCTOR 




642 


TUNNEL 


R ES 1 ST 1 V 




72 4 


INPUT 


C AP AC 1 T 








ACCESS 


358 


1 49 


1 NTERPL A 


SHUNT 




358 


CORPUSC 


F 1 X 




705 


FLAKE 



3 25 

544 

728 

780 

98 3 

3 3 

326 

6 1 0 

766 

327 

37 

1 45 

235 

3 28 

36 3 

7 4 4 

296 

3 0 1 

3 29 

403 

8 4 4 

330 

3 3 1 

77 1 

98 

332 

53 0 

3 3 3 

490 

596 

3 3 4 

3 35 

5 1 5 

826 

987 

2 25 

3 36 

46 9 

45 

337 

3 8 4 

26 2 

338 

53 1 

54 

72 

27 1 

3 39 

538 

6 4 3 

6 73 

759 

23 3 

294 

3 1 2 

3 40 

4 1 2 

58 I 

739 

53 

1 02 

1 3 4 

1 35 

25 1 

3 4 1 

48 4 

79 1 

973 



93 



358 


736 


MAJOR 


3 7 3 


3 59 


3 59 


CORRECT 




3 fa 0 


2 1 9 


P 1 ezoele 






36 0 


CRYSTAL 






382 


EXC 1 TON 


3 74 




648 


V 1 8R A T 






785 


SHEAR 


3 7 5 


3 6 1 


1 04 


FILAMENT 






36 1 


CURRENT 






39 3 


J N D U C T 1 


376 




603 


return 






722 


1 NDUC 






97 1 


EDO 


3 7 7 


36 2 


6 


after NOO 






1 56 


1 ONOSPHE 






362 


DAYTIME 






372 


0 1 URN AL 






46 3 


STATION 






b09 


SEASON 






627 


SUMMER 






fa52 


winter 






751 


NIGHT 


378 


36 3 


24 


AUTOM A T 1 






3 fa 3 


DECIMAL 






4 4 1 


reading 






665 


DINAR 






fa 8 7 


digit 






940 


tape 






96 2 


add 


3 79 




9 66 


B 1 T 






968 


COD 




3 6 4 


235 


R AD 1 A T 1 0 






328 


WAVELENG 






364 


DEC 1 MET 




3 65 


fa 5 


CRYOTRON 


38 0 




29 1 


SUPERCON 






3 6 5 


OEPOS 1 T 






866 


FILM 


38 1 


3 66 


200 


OBSER vat 






26 0 


S ATELL 1 T 






366 


detonat 






383 


EXPLOS 1 






4 1 5 


NUCLEAR 






734 


LOCAT 






944 


TRAP 


382 


367 


36 7 


DEVELOP 




3 6 8 


36 8 


0 1 SP ER S 




369 


369 


D 1 SPLAC 


383 


3 70 


59 


CON V EN T 1 






6 7 


DEMODULA 






27 0 


SINUSOID 






29 3 


SYNCH RON 






3 7 0 


DISTORT 


384 




67 1 


f'LASS 


3 7 1 


1 1 4 


SEOM AG N£ 






1 8 4 


morpholo 






3 7 1 


disturb 






7 1 2 


giant 






733 


local 






762 


POLAR 




3 72 


6 


afternoo 


3 86 




1 56 


1 ONOSPHE 






29 7 


temporal 






324 


VAR 1 A r 1 0 






362 


daytime 


386 




3 72 


diurnal 




6 0 9 


SEASON 






6 7 9 


DAILY 




o 


73 3 


local 




ERIC 


76 8 
797 


Quiet 

south 


387 




882 


HOUR 





387 


1 2 3 


HYDRODYN 


DOPPLER 




22 7 


PRESSURE 


FARADAY 




387 


GASEOUS 


SPUTNIK 




545 


HELIUM 


SHIFT 




587 


PLASMA 






766 


PROBE 


DYNAMIC 










388 


1 79 


M 1 LL 1 met 


LUM 1 NE SC 




) 80 


M 1 L L 1 K) 1 C 


PHOTOCON 




289 


SUBM 1 LL 1 


ELECTRO 




388 


G E N E R A T 






849 


D 1 V 1 


NONREC 1 P 




922 


RISE 


SYNTHES 1 








ELEMENT 


389 


3 89 


GEOMETR 


CONF 1 GUR 


390 


1 59 


1 SOTHERM 


1 NTERNAL 




390 


HEATING 


TRANS 1 ST 








collect 


39 1 


99 


EXT 1 NCT I 


EMITTER 




39 1 


1 NC 1 DE N 


AMPL 1 F 




444 


REFLECT 


CLASS 




576 


OBL 1 QU 


SHUNT 




9 3 1 


SLIT 


BIAS 










39 2 


392 


1 ND 1 C AT 


CONVENT 1 








RESPONSE 


39 3 


1 34 


1 NDUCTAN 


SELECT 1 V 




36 1 


CURRENT 


EQUAL 1 Z 




393 


1 NDUCT 1 


tele V 1 S 








DESIGN 


39 4 


394 


INERTIA 






62 1 


STATIC 


G F 0 M A G N E 








1 ON 1 Z A T 1 


39 5 


72 


DIFFRACT 


1 ONOSPHE 




86 


ELECTRUM 


latitude 




27 1 


SOLUTION 


EQUATOR 




395 


1 N F 1 N 1 T 


LAYER 




5 1 3 


CONVEX 


QUIET 




6 06 


SCALAR 






6 07 


SCREEN 


CALCULAT 




699 


EXACT 


FUNCTION 




759 


PLANE 


E VALUAT 




786 


SHEET 






967 


BOD 


EXC 1 TAT 1 








CSC 1 LLAT 


3 9 6 


396 


INITIAL 


STANDI NG 








EXCITED 


397 


397 


1 NSULAT 


PLASMA 




743 


METAL 


CA V 1 T 










398 


232 


PROPORT 1 


1 NTER ACT 




398 


INVERSE 


CRYSTAL 








EXC 1 TON 


399 


1 2 


AN 1 SOTRO 


LATTICE 




70 


D 1 ELECTR 


V 1 BR AT 




86 


ELECTROM 


THEOR 




399 


1 SOTROP 






635 


TENSDR 


detonat 








EXPLOS 1 


40 0 


200 


OBSERVAT 


NUCLEAR 




23 5 


R AD 1 AT 1 0 


ISLAND 




400 


JUP 1 TER 






586 


PLANET 


S A TE LL 1 T 




77 1 


RADIO 


BEARING 








DOPPLER 


40 1 


44 


COLLI i 


Faraday 




1 2 1 


HOMOGLNE 


REFLECT 




246 


RELAT f V 1 


DENS 1 T 




31 6 


TR ANSPOr 


FADING 




401 


KINETIC 


R 0 T A T 




532 


ENERG Y 






7 08 


FORCE 


magnet t C 




807 


THEOR 


FERRITE 








MAGNET 1 


402 


3 8 2 


EXC 1 TON 


CORE 




402 


LATTICE 


FUND AMEN 


403 


29 3 


S YNCHRON 


TRANSFOR 




329 


AM8 1 E N T 


FOURIER 




357 


CONVERT 


STEADY 




403 


L E AO 1 N G 


UNIQUE 










404 


89 


engineer 


D 1 $CH ARG 




404 


machine 



3 7 3 

3 84 

460 

7 88 

374 

1 65 

2 1 5 

375 

1 9 5 

294 

3 76 

55 

1 47 

3 1 4 

348 

377 

49 1 

67 1 

79 1 

8 35 

59 

257 

26 3 

378 

47 1 

520 

1 1 4 

1 53 

1 56 

1 62 

379 

728 

768 

35 

1 09 

380 

94 

203 

283 

3 8 1 

58 7 

668 

1 44 

360 

382 

402 

6 48 

8 07 

366 

383 

4 1 5 

553 

260 

337 

37 3 

384 

444 

5 1 9 

537 

777 

) 66 

3 85 

405 

8 45 

1 ^ 0 

3 1 2 

386 

622 

643 

76 



94 




4 04 


435 


PROGRAM 


4 1 6 


7 55 




4 B 4 


ACCESS 




850 




5 1 1 


C 0 M P U T 








632 


system 


4 1 7 


1 89 




fa B 7 


DIGIT 




257 




8 54 


DRUM 




256 




938 


STOR 




3 1 3 










4 1 7 


405 


1 03 


ferromag 




462 




1 66 


MAGNET 1 C 




7 1 8 




385 


FERRITE 




922 




405 


MAGNET 1 




937 










942 


4 06 


4 4 


COLL 1 S 1 0 








1 00 


EXTRAORD 


1 1 8 


242 




1 23 


HYDROO YN 




4 1 8 




23 1 


PROPAG A T 








4 06 


MAGNETO 


4 1 9 


292 




480 


VELOC 1 T 




4 1 9 




5B7 


PLASMA 




7 04 




7 25 


IONIC 




7 1 9 




7 26 


1 ON 1 Z 




9 1 5 




876 


GYRO 




978 


4 07 


67 


demddula 


42 0 


420 




4 07 


MODUL A T 




53 0 










7 26 


4 08 


4 08 


MOLECUL 








fa 57 


AMMON 


42 1 


3 










343 


4 09 


409 


MULT 1 P L 




42 1 










698 


4 1 0 


2 03 


DSC 1 LL A T 








255 


RESONATO 


422 


422 




4 \ 0 


NATURAL 












423 


423 


4 1 1 


224 


POSITIVE 




461 




4 1 1 


NEGAT I V 




491 










639 


4 1 2 


4 


ADM 1 TT An 




8 1 4 




233 


QU AOR 1 P 0 




827 




294 


SYNTHES 1 




8 75 




298 


terminal 








3 1 1 


TRANSFER 


42 4 


2 2 7 




3 , 0 


CANONIC 




260 




-r 1 2 


NETWORK 




4 1 6 




4 9 8 


8R ANCH 




4 2 4 




555 


1 TERAT 




540 




5 5 6 


ladder 




5 44 




58 1 


PASS 1 V 




755 




739 


M ATR 1 




850 


4 1 3 


4 4 


coll 1 S 1 0 


425 


1 2 




87 


ELECTRON 




70 




1 3 1 


1 NCOHERE 




425 




4 1 3 


neutral 








52 4 


D \ FFUS 


426 


426 




58 7 


PLASMA 








62 1 


STATIC 


42 7 


427 




7 26 


i ON t Z 




570 




976 


GAS 








979 


1 0 N 


428 


42B 


4 1 4 


4 1 4 


neutron 


429 


429 




5 1 5 


COSMIC 




625 




59 0 


proton 




764 




68 1 


OECA Y 




825 




723 


INNER 




836 




83 4 


SELT 




950 




98 7 


RAY 












430 


99 


4 1 5 


234 


OUAORUPO 




430 




254 


PESONANC 




694 




366 


DETONAT 








3 B3 


EXPLOS 1 


43 1 


43 1 




4 1 5 


NUCLEAR 








553 


ISLAND 


432 


262 










432 


4 1 6 


2 1 


atmosphe 




532 




C 1 


ECCENTR 1 








260 


SATELL 1 T 


433 


29 




4 1 6 


orbital 




433 




42 4 


perigee 








693 


earth 


434 


87 



ORBIT 


43 4 


434 


PROFILE 


DRAG 




48 1 


V 1 RTUAL 






519 


DENS 1 T 


MUL T 1 ST A 




544 


HEIGHT 


RESPONSE 








RESTR 1 CT 


135 


89 


ENGINEER 


TRANS 1 EN 




404 


MACHINE 


OVERSHO 




435 


program 


STAGGER 




5 1 1 


C 0 M P U T 


IDEAL 








RISE 


43 6 


252 


RESISTOR 


STEP 




436 


PROTECT 


TIME 




792 


S 1 L 1 C 


REFLECTO 


43 7 


43 7 


QUALITY 


P AR ABOL 




9 1 4 


PULL 


SUPERREG 


438 


438 


RADIANT 


PAR AMET 




567 


METEOR 


F 1 GUR 




6 1 2 


SHOWER 


IMAGE 




8 1 0 


TRAIL 


PUMP 








1 DL 


439 


1 35 


INDUCTOR 






23 7 


REACT ANC 


PARTIAL 




25 1 


R ES 1 ST 1 V 


ECL 1 PS 




299 


TERM 1 NAT 


1 ON 1 Z 




43 9 


RE ACT 1 V 






49 B 


BRANCH 


AD 1 ABAT 1 




990 


ROW 


central 








PASSAGE 


4 4 0 


3 45 


CIRCUIT 


ERUPT 




4 4 0 


REACTOR 






45 0 


SATUR A8 


PATTERN 




451 


S ATUR AT 






579 


OUTPUT 


P E rj T 0 D E 




589 


PRESET 


ST AB 1 L 1 




8 4 5 


CORE 


AMPL 1 F 




95 1 


VOLT 


TR 1 ODE 








VALVE 


4 4 1 


24 


AUTOMAT I 


A N DD 




1 36 


1 N F 0 R M A T 


GRID 




364 


DECIMAL 






4 4 1 


R E A D 1 rJ G 


PRESSURE 




589 


PRESET 


SATELL 1 T 




665 


8 1 N A R 


ORBITAL 




8 54 


DRUM 


PERIGEE 




9 1 9 


READ 


GRA V 1 T 




940 


TAPE 


HEIGHT 




959 


WRIT 


ORBIT 




9eB 


COD 


DRAG 










442 


4 42 


R ECOGN 1 


AN 1 SOTRO 








D 1 ELECTR 


4 43 


443 


RECOVER 


P ER ME A8 




942 


TIME 


PERSIST 


4 4 4 


99 


EXT 1 NCT 1 






259 


ROUGHNES 


P ER T UR8 




384 


FARADAY 


MOTION 




391 


1 NCI DEN 






4 4 4 


REFLECT 


PICTURE 




4 8 1 


VIRTUAL 






576 


OB L 1 QU 


PLASTIC 








STRESS 


4 4 5 


236 


R AO 1 0 S ON 


PRINT 




4 45 


REFRACT 


ADHE 




505 


CL 1 MAT 


8 OND 




72 1 


1 NOEX 


VIEW 










446 


307 


THYR ATRO 


ext 1 NCT 1 




446 


REGULAT 


POLAR 1 Z 




4 68 


S U P P L ; E 


ELL 1 P 




579 


OUTPUT 






763 


POWER 


PREDICT 




B52 


DR I V 






95 1 


VOLT 


SECONDAR 




995 


TUN 


PRIMARY 




998 


DC 


ENERGY 










447 


4 4 7 


RELEASE 


S ANDW 1 DT 




6 05 


ROCKET 


PRODUCT 




672 


CLOUD 


ELECTRON 


4 4 8 


4 48 


R EVERS A 



95 



4 4 8 


692 


DRIFT 


46 3 


4 4 9 


4 4 9 


REVERSE 


46 4 




59 9 


RECT 1 F 






781 


SEL EN 






792 


S 1 L 1 C 


46 5 


4 5 0 


3 07 


THYR ATRO 


46 6 




4 4 0 


REACTOR 






45C 


S ATUR AB 






45 1 


S ATURAT 






552 


1 NVERT 






8 4 J5 


CORE 


46 7 


451 


1 66 


MAGNETIC 






4 4 0 


REACTOR 






450 


S ATURA8 






45 1 


S A T LI R A T 




452 


63 


CORHUGAT 






99 


EX T 1 NCT 1 






1 3 1 


1 NCOHERE 






452 


SCATTER 


468 




494 


ASPECT 






776 


rigid 




453 


4 5 3 


sect ion 






b77 


CROSS 




4 5 4 


1 5 


APPROX 1 M 






257 


response 






4 54 


SEGMENT 






559 


LINEAR 


46 9 




832 


8 ASE 




455 


356 


control 






455 


SENSING 




4 56 


2 1 8 


PhOTOSEN 






3 45 


CIRCUIT 






4 56 


shaping 


470 




766 


PULSE 


4 7 1 


4 5 7 


457 


similar 




4 5 8 


4 58 


S 1 MUL AT 




459 


80 


D 1 ST R I 8 U 






459 


SPATIAL 


472 


46 0 


1 7 


AR T 1 F 1 C 1 






260 


SATELL 1 T 






373 


DOPPLER 


473 




460 


SPUTNIK 






537 


fading 






755 


ORBIT 


4 7 4 


46 1 


48 


COMpENS A 






1 01 


FEED8 ACK 






3 0 1 


THERM 1 ST 






423 


pentode 






4 6 1 


S T A B 1 L 1 


475 




468 


S U P P L 1 E 






499 


8 R 1 DGE 






543 


heater 


476 




875 


grid 






935 


ST A8 


477 


462 


49 1 


AMPL 1 F 






572 


narrow 






633 


TABULA 






691 


00U8L 






83 1 


8 A ND 






9 05 


PASS 






995 


TUN 




463 


1 1 3 


Geograph 


478 




1 20 


hem 1 SPHE 






1 56 


1 ONOSPHE 






362 


daytime 






4 63 


STATION 






752 


north 


479 




797 


SOUTH 






822 


WORLD 




ERJC 

















YEAR 


4 8 0 


4 06 


MAGNETO 






4 8 0 


VELOC 1 T 


STR A 1 GH 




571 


MOVING 


LINE 




6 92 


DRIFT 






805 


SURGE 


STR AT 1 F 










4 8 1 


1 4 


apparent 


NONSEASO 




1 56 


1 ONOSPHE 


SUNRISE 




272 


SOUNDING 


SUNSET 




434 


profile 


LAYER 




44 4 


REFLECT 


NIGHT 




4 8 1 


VIRTUAL 






54 4 


HEIGHT 


ACTIVITY 




6 4 5 


valley 


ST AT 3 ST 1 




86 3 


FALL 


central 








SUNSPOT 


4 82 


3 7 


C ENT 1 met 


NUMBER 




4 8 2 


V 1 S 1 8LE 


CYCLE 








SOLAR 


4 8 3 


483 


WINDING 


YEAR 




8 4 5 


CORE 


thyratro 


4 8 4 


1 3 6 


1 NFORM AT 


REGULAT 




3 4 1 


C AP AC 1 T 


ST A8 1 L 1 




4 04 


MACHINE 


SUPPL 1 E 




4 04 


ACCESS 


OUTPUT 




595 


RANDOM 


POWER 




7 3 5 


LOGIC 


VOLT 




742 


MEMOR 


TUN 




85 4 


DRUM 


DC 




938 


S TOR 






966 


8 1 T 


CORRUGAT 








1 MPERFEC 


4 85 


4 8 5 


ACCURA 


ROUGHNES 








8 AR R 1 E R 


4 86 


4 8 6 


ACTION 


SURFACE 




73 7 


MASER 


CONVEX 




795 


SOLID 


ROCK 










4 87 


1 28 


1 MPEDANC 


s yumetr 




4 8 7 


ACTIVE 


RESPONSE 


488 


1 08 


FREQUENC 


EQUAL 1 Z 




48 8 


ADJUST 


TELEV 1 S 




49 7 


8 ALANC 


COLOUR 




54 3 


HEATER 


SAND 




8 1 4 


VALVE 


THERMAL 


4 89 


1 8 4 


MORPHOLO 


HE AT 




4 8 9 


AOV ANC 


LAW 




529 


DYNAMO 


L 1 GHTN 1 N 


49 0 


75 


D 1 RECT 1 V 


THUNDER 




1 4 5 


1 NTERFER 


FLASH 




25 3 


R ESO LUT 1 






295 


TELESCOP 


L 0 NG 1 T U D 




3 3 3 


antenna 


MAGNETOS 




4 9 0 


AERIAL 


TORSION 




523 


D 1 AMET 


F 1 LT 




596 


R ECE 1 V 






6 00 


R ESO L V 


TRANSIT 




66 0 


ARRAY 


TIME 




6 6 1 


ASTRO 


T R A N S U 1 


49 1 


29 


8 AND W 1 DT 






4 8 


COMPENS A 


8 1 ST A8 L E 




292 


SUPERR EG 


8L0CKI NG 




377 


EMITTER 


MONOSTAB 




423 


PENTODE 


MULT 1 V 1 B 




46 2 


STA6GER 


TRANS 1 ST 




49 1 


AMPL 1 F 


CIRCUIT 




497 


8 ALANC 


TRIGGER 




7 0 4 


F 1 GUR 


PULSE 




838 


CHOP 


FLIP 




87 4 


GAIN 






9 1 5 


PUMP 


FREQUENC 




978 


I DL 


TUNABLE 








SAND 


492 


1 96 


NONSEASO 


rang 




492 


ANOM AL 






93 0 


SKIN 


UNIFORM 








STATIC 


493 


1 56 


1 ONOSPHE 






493 


ARCTIC 



960 

4 6 4 

880 

465 

1 96 

466 

6 28 

7 28 

751 

2 

2 85 

3 43 

467 

5 75 

6 78 

794 

960 

3 07 

4 46 

46 1 

4 68 

579 

7 63 

951 

9 95 

9 98 

63 

1 29 

259 

3 36 

469 

5 1 3 

9 2 3 

4 70 

257 

378 

4 7 1 

508 

8 3 1 

472 

8 78 

9 8 1 

1 63 

4 73 

7 06 

1 6 4 

1 70 

474 

867 

4 75 

942 

4 76 

32 

3 3 

1 83 

1 90 

3 1 4 

3 45 

477 

766 

869 

1 08 

4 7 8 

83 1 

9 1 7 

4 *'9 

6 2 1 



507 



49 4 


1 57 


1 RKEGUL A 






266 


SENS 1 T i V 






4 52 


SCATTER 






49 4 


ASPECT 






6 55 


ALIGN 


5 0 8 




858 


ECHO 




495 


1 9 1 


NITROGEN 


509 




2 40 


RECOMB 1 N 






495 


ATOMIC 


5 1 0 




580 


OXYGEN 






829 


ATOM 


5 1 1 


496 


269 


S 1 Kl U L T A N 






496 


AURORA 


5 1 2 




553 


1 SLAND 






96 5 


8 AY 






996 


ZON 




497 


28 


8 ALANCED 






345 


CIRCUIT 






48B 


ADJUST 






49 1 


AMPL 1 F 






4 9 7 


8 ALANC 






543 


heater 






552 


INVERT 


5 1 3 




579 


OUTPUT 






8 1 4 


VALVE 






95 1 


VOLT 






9 98 


DC 




498 


1 2 8 


1 MPED ANC 






1 38 


1 N SERT 1 0 






233 


QU ADR 1 PO 






299 


T ER M 1 N A T 






4 1 2 


NETWORK 


5 1 4 




439 


RE ACT 1 V 






498 


BRANCH 






556 


LADDER 






7 1 9 


IMAGE 






86 7 


F 1 LT 






905 


PASS 


5 1 5 


499 


1 0 1 


FEED8ACK 






263 


SELECT 1 V 






46 1 


S T AB 1 L 1 






499 


8R 1 DGE 






855 


DUAL 




500 


80 


D 1 ST R 1 8 U 






20 0 


08SERVAT 






235 


RAD 1 AT 1 0 






500 


8R 1 GHT 






5 1 4 


CORONA 






573 


NE8ULA 






77 1 


RADIO 


5 1 6 




828 


AREA 






964 


ARC 






993 


SUN 




50 1 


252 


RESISTOR 






50 1 


C AR80N 






535 


extrem 






866 


film 




502 


1 28 


1 MPED A NC 






233 


OU ADR 1 PO 


5 1 7 




502 


C A SC AO 






905 


PASS 


5 1 8 


603 


503 


CATHOD 


5 1 9 




82 7 


A N OD 






842 


COLO 




504 


223 


POSITION 






50 4 


CENTRE 




505 


4 4 5 


Refract 






505 


CL 1 MAT 




506 


279 


SPONTANE 






506 


COHERE 


52 0 




CLOSED 


52 0 


263 


SELECT 1 V 


SYSTEM 




28 1 


SPURIOUS 


SERVO 




3 08 


TOLER ANC 


LOOP 




378 


EQUAL I 2 






520 


DESIGN 


T E U E V 1 S 




663 


BASIC 


COLOUR 




69 1 


DOUBL 






738 


MATCH 


COLUMN 


52 1 


52 1 


DETECT 


C0M8 1 N 
LINEAR 


5 2 2 


522 


DEV 1 AT 


COMPOS 


523 


1 46 


1 NTERFER 






295 


TELESCOP 


ANALOGUE 




4 9 0 


AERIAL 


ENGINEER 




5 23 


D 1 AMET 


INFORMAT 




568 


M 1 R R 0 


MACHINE 




66 1 


ASTRO 


PROGRAM 
C OMPUT 




7 4 4 


METRE 


DIGIT 


52 4 


2 2 


AT T ACHME 


DRUM 




8 V 


ELECTRON 


READ 




1 53 


1 ON 1 Z AT 1 


TAPE 




4 1 3 


NEUTRAL 






5 1 9 


DENS 1 T 


ASYMPTOT 




524 


D 1 FF US 


DIFFRACT 




726 


1 ON 1 Z 


CONDUCT 




979 


1 ON 


1 NF 1 N 1 T 
SURFACE 
CONVEX 


525 


525 


DIRECT 


C YL 1 nd 
PLANE 


526 


526 


D 1 S T A N 


BOD 


527 


527 


D 1 VERG 


R AD 1 AT 1 0 
BRIGHT 


528 


528 


D 0 M M N 


CORONA 


529 


1 1 4 


GEOM AGNE 


ERUPT 




1 1 7 


GRAD 1 E N T 


SOLAR 




1 56 


lONOSPHE 


SUN 




266 


SEM 1 D 1 UR 






4 8 9 


ADVANC 


1 NT ENS 1 T 




529 


DYNAMO 


1 NT ERPlA 




707 


FLUID 


terrestr 




768 


QUIET 


BALLOON 




807 


THEOr 


NEUTRON 




89 4 


MAIN 


COSMIC 




939 


TAIL 


ORIGIN 




994 


T 1 D 


PROTON 

flare 


5 3 0 


33 2 


ANNULAR 


BELT 




420 


PARTIAL 


FLUX 




530 


ECL 1 PS 


RAY 




794 


SOLAR 


ACOUSTIC 


53 1 


2 6 2 


SECO ND AR 


SPHER 1 CA 




2 79 


SPONTANE 


conduct 




338 


8 0 M 8 A R D 


CONVEX 




53 1 


EM 1 SS 1 


C YL 1 ND 




623 


ST 1 MUL 


SPHERE 




634 


TARGET 


axial 

CONIC 




860 


EMIT 


SHEET 


53 2 


93 


EXCHANGE 


AXES 




1 44 


interact 


DEFORM 




1 82 


MOMENTUM 




26 2 


secondar 






35 1 


CONSER V 


DEGREE 




4 0 1 


kinetic 






432 


PR 1 MARY 


concentr 

NONSEASO 




532 


ENERGY 


FARADAY 

PROFILE 


533 


533 


EX AM 1 N 


DENS 1 T 
D 1 FFUS 


534 


534 


EXTEND 


valley 


53 5 


2 0 4 


CSC 1 LLOG 


SCALE 




5 0 1 


CARBON 


UPPER 




535 


EXTREM 


FALL 




6 04 


RIPPLE 






9 36 


STAB 


ATTENUAT 
M 1 CROM 1 N 




982 


LOW 



507 

632 

782 

892 

47 1 

508 

509 

5 1 0 

559 

5 1 1 

1 0 

89 

1 36 

4 04 

435 

5 1 2 

687 

854 

9 1 9 

9 40 

20 

72 

350 

395 

469 

5 1 3 

5 1 6 

7 59 

967 

235 

500 

5 1 4 

698 

794 

9 93 

1 43 

1 49 

3 00 

335 

4 1 4 

5 1 5 

578 

59 0 

7 05 

834 

87 1 

987 

1 

277 

350 

5 1 3 

5 1 6 

6 1 7 

662 

673 

786 

830 

5 1 7 

5 1 8 

52 

1 96 

384 

43 4 

5 1 9 

524 

6 4 5 

780 

8 1 3 

863 

23 

1 75 



97 



5 3 6 


536 


FACTOR 


553 


53 7 


1 5 7 

3 8 4 

4 6 0 


1 RREGUL A 

FARADAY 

SPUTNIK 






53 7 
6 1 3 


FADING 

SIGNAL 


554 




9 56 


W 1 ND 


555 


538 


2 0 


AS YMPT OT 






1 72 

2 7 1 
284 

3 39 

5 3 8 

6 4 3 
b 4 7 
796 


MATHEM at 
SOLUT 1 DN 
ST AT 1 ONA 

boundar . 
finite 

UNIQUE 

VECTOR 

SOLVE 


5 56 


5 39 


53 9 
5 5 7 


flight 

LAUNCH 






5 6 3 

6 05 


ME ASUR 
ROCKET 


55 7 


54 0 


4 2 4 

5 4 0 


PERIGEE 
Gr A V 1 T 






6 93 


earth 


558 


541 


5 4 1 


Ground 


559 


542 


2 4 6 
2 8 4 
5 4 2 


REL AT 1 V 1 
ST at 1 DNA 
Growth 




5 4 3 


2 8 

4 6 1 


BALANCED 
ST A8 1 L 1 


56 0 




488 
4 9 7 


ADJUST 
8 ALANC 


56 1 




5 4 3 
B 1 4 


heater 

VALVE 


56 2 




95 1 


VOLT 


563 


54 4 


3 1 

1 96 
280 
3 2 5 
424 
43 4 


B 1 FURC AT 
NONSEASO 
SPORADIC 

vertical 

PERIGEE 

PROFILE 






4 8 1 

5 4 4 
b28 
75 1 
8 1 3 


VIRTUAL 

Height 

SUNSET 

night 

UPPER 


56 4 


545 


1 9 1 


nitrogen 






2 2 7 
387 
54 5 
587 
76 5 


PRESSURE 

Gaseous 

Helium 

PLASMA 

PROBE 


565 


546 


59 


Con VENT 1 






1 6 0 
5 46 


JUNCT 1 ON 
H Y B R 1 D 


56 6 


547 


547 


1 DENT 1 


56 7 


5 48 


5 4 8 


1 MPACT 




549 


5 4 9 


1 MPULS 




550 


550 


1 NCL 1 N 






755 


Orbit 


56 8 


55 1 


2 4 6 
55 1 


RELAT 1 V 1 
INJECT 




552 


4 5 0 

4 9 7 

5 5 2 
579 
852 
908 


SATUR AB 

BALANC 

INVERT 

Output 
DR 1 V 
PH AS 


569 



o 




EXPLOS 1 








NUCLEAR 


570 


4 2 7 


PERTURB 


AURORA 




570 


MOTION 


ISLAND 










57 1 


3 8 


CERENKOV 


1 SOL AT 




2 08 


PARTICLE 






2 4 6 


RELAT I V 1 


NETWORK 




4 8 0 


VELOC 1 T 


1 TER AT 




5 7 1 


MOVING 


1 MPEDANC 


572 


2 9 


B AND W 1 D T 


1 NSER T 1 0 




1 0 8 


FREQUENC 


T ERM 1 NAT 




263 


SELECT 1 V 


NETWORK 




4 6 2 


STAGGER 


BRANCH 




5 7 2 


NARROW 


LADDER 




6 7 6 


COUPL 


IMAGE 




8 3 1 


BAND 


MATCH 








F 1 LT 


5 73 


78 


D 1 SCRETE 


PASS 




1 1 1 


GALACTIC 






2 0 0 


OBSER VAT 


S ATELL 1 T 




2 3 5 


R AD 1 A T 1 0 


FLIGHT 




5 0 0 


BRIGHT 


LAUNCH 




5 7 3 


NEBULA 


ROCKET 




6 1 5 


SOURCE 






6 6 1 


ASTRO 


LENGTH 




7 09 


GALAX 






7 7 1 


RADIO 


T R ANS 1 EN 




96 4 


ARC 


SEGMENT 








COMB I N 


574 


5 7 4 


NORMAL 


linear 








M ATR 1 


575 


46 7 


S U M S P 0 T 






5 7 8 


NUMBER 


LIQUID 










5 76 


99 


EXT 1 NCT 1 


MAGNET 




3 9 1 


1 NC 1 DEN 






4 4 4 


REFLECT 


MATTER 




576 


OBL 1 QU 


MAGNETOM 


577 


99 


EX T 1 NCT 1 


R A D 1 0 S ON 




577 


ORIENT 


ROUGHNES 








Flight 


5 78 


5 1 5 


COSMIC 


ME ASUR 




5 7 8 


ORIGIN 


ROCKET 








PROBE 


579 


8 3 


EFF 1 C 1 EN 






3 0 7 


THYR ATRO 


AFTERNOO 




4 4 0 


REACTOR 


MEDIAN 




4 4 6 


REGULAT 


CYCLE 




4 6 8 


SUPP L I E 


DAILY 




4 9 7 


BALANC 


month 




55 2 


INVERT 


YEAR 




5 7 9 


OUTPUT 






7 2 4 


INPUT 


AN 1 SOTRO 




8 5 2 


DR 1 V 


0 1 EL EC T R 




96 2 


ADD 


ELECTROM 








MED 1 UM 


58 0 


2 1 


ATMOSPHE 


MEDIA 




42 


COEFF 1 C 1 


WAVE 




52 


CONCENTR 






1 06 


FORM AT 1 0 


D 1 SCHARG 




1 9 1 


NITROGEN 


MERCUR 




238 


REACTION 






2 4 0 


R ECO MB 1 N 


TURBULEN 




4 9 5 


ATOMIC 


RADI ANT 




5 8 0 


OXYGEN 


meteor 




7 1 0 


GASES 


SHOWER 




863 


FALL 


trail 








ECHO 


58 1 


4 


ADM 1 TTAN 






1 95 


NONR EC 1 P 


TELESCDP 




233 


QUADR IPO 


D 1 AME T 




239 


R EC 1 P R DC 


MIRROR 




298 


TERMINAL 


L DC A T 




3 4 0 


C A 0 N I c 






4 1 2 


network 


INTERACT 




58 1 


P A SS 1 V 


MAGNETIC 




739 


M ATR 1 


MAGNET 1 S 








MOMENT 


58 2 


1 78 


midnight 


FIELD 




285 


ST AT 1 ST 1 


1 NDUC 




3 4 3 


central 


SPIN 




582 


PERIOD 



3 83 

4 1 5 

4 96 

553 

554 

4 1 2 

555 

1 28 

1 38 

299 

4 1 2 

498 

556 

7 1 9 

7 38 

86 7 

9 05 

260 

539 

557 

6 05 

558 

3 1 3 

4 5 4 

5 1 0 

559 

7 39 

56 0 

56 1 

562 

1 68 

236 

2 59 

539 

5 6 3 

6 05 

7 65 

6 

56 4 

6 78 

6 79 

7 49 

9 6 0 

1 2 

70 

86 

565 

7 4 1 

954 

76 

566 

3 1 9 

438 

56 7 

6 1 2 

8 1 0 

8 58 

295 

523 

568 

734 

I 4 4 

1 66 

I 6 7 

569 

7 03 

7 22 

934 



98 










597 


9 4 0 


5B3 


583 


PHOTON 












598 


589 


S 8 4 


584 


PHYSIC 




598 


b85 


585 


planar’ 


599 


1 1 6 










4 4 9 


586 


400 


JUPITER 




599 




586 


PLANET 




763 










78 1 


587 


b 6 


CYCLOTRO 




792 




1 23 


H Y DR OD Y N 




8 4 4 




1 6 4 


LONG 1 TUD 








283 


STAND 1 NG 


6 0 0 


253 




38 1 


excited 




490 




3 87 


GASEOUS 




6 00 




4 06 


MAGNETO 




763 




4 1 3 


NEUTRAL 








54 5 


helium 


6 0 1 


4 7 




587 


PLASMA 




1 08 




707 


FLUID 




2 0 3 




765 


PROBE 




6 0 1 


588 


3 56 


control 


6 02 


6 02 




588 


PRECIS 








890 


LOCK 


6 03 


1 6 3 










36 1 


589 


3 56 


CONTROL 




6 0 3 




4 4 0 


REACTOR 




6 26 




4 4 1 


READING 








589 


PRESET 


6 0 4 


53 5 




598 


RECT AN 




6 0 4 




b 1 3 


SI GNAL 








8 0 0 


SPEED 


6 03 


1 4 0 




8 6 4 


FAST 




4 4 7 




8 9 0 


LOCK 




53 9 




966 


8 1 T 




557 




977 


GAT 




56 3 










6 05 


590 


208 


PARTICLE 








4 1 4 


NEUTRON 


6 06 


20 




5 1 5 


COSMIC 




4 0 




59 0 


PROTON 




72 




723 


1 N N E R 




2 7 1 




757 


OUTER 




278 




83 4 


8E LT 




395 




9 44 


TRAP 




6 06 




987 


RAY 




6 4 3 










6 4 7 


5 9 1 


1 1 4 


geomagne 




776 




1 25 


HYOROMAG 








1 76 


MICROPUL 


6 07 


72 




59 1 


PULSAT 




86 




7 1 2 


giant 




2 7 1 




8 03 


STORM 




350 




96 5 


BAY 




395 










6 0 7 


59 2 


592 


RADIAL 




759 




770 


RADII 




93 1 










954 


593 


593 


RAD 1 AT 












608 


1 3 


59 4 


594 


RADIUS 




608 




967 


BOO 












6 09 


1 56 


595 


286 


STOCHAST 




1 96 




4 8 4 


ACCESS 




2 9 7 




595 


Random 




32 4 










362 


596 


2 1 7 


photogr A 




3 72 




29 5 


telescop 




6 09 




3 33 


ANTENNA 




6 2 7 




49 0 


aerial 




6 52 




596 


RECE 1 V 




678 










6 7 9 


597 


1 8 


ASSOC 1 A T 




7 2 8 




1 50 


1 NTER VAL 




7 4 9 




1 55 


1 ONOSOND 




96 0 




2 1 7 


photogra 








280 


SPORAD 1 C 


6 1 0 


1 83 




597 


RECORD 




2 8 1 




6 4 5 


VALLEY 




3 26 




7 1 2 


Giant 




6 1 0 




8 77 


head 




636 



TAPE 


6 1 0 


683 


DELAY 






766 


PULSE 


PRESET 




793 


S t N G L 


R ECT AN 




86 4 


FAST 


GERMAN 1 U 


6 1 1 


6 1 1 


SERIES 


REVERSE 








RECT 1 F 


6 1 2 


4 3 8 


RADIANT 


POWER 




66 7 


METEOR 


SELEN 




6 1 2 


SHOWER 


S 1 L 1 C 




6 4 9 


VISUAL 


COOL 




8 1 0 


TRAIL 


RESOLUT 1 


6 1 3 


53 7 


FADING 


aerial 




589 


PRESET 


RE SOLV 




6 1 3 


SIGNAL 


POWER 




978 


1 OL 






984 


M 1 X 


C 0 M P A T 1 8 








FREQUENC 


6 1 4 


6 1 4 


SILVER 


OSC 1 LL AT 




836 


BONO 


RE SONA 










6 1 5 


78 


DISCRETE 


RETARD 




1 1 1 


GALACTIC 






1 45 


! NTERFER 


L 1 GH TN 1 N 




5 73 


NEBULA 


CURRENT 




6 1 5 


SOURCE 


RETURN 




629 


SURVEY 


STROKE 










6 1 6 


286 


STOCHAST 


EX TREM 




6 1 6 


SPECTR 


RIPPLE 










6 1 7 


86 


ELECTROM 


1 NSTRUME 




350 


CONDUCT 


RELEASE 




5 1 6 


C YL 1 ND 


flight 




6 1 7 


SPHERE 


LAUNCH 




807 


THEOR 


ME ASUR 




830 


AXES 


ROCKET 










6 1 8 


6 1 8 


SP L ) TT 


ASYMPTOT 








C 1 R CUL AR 


6 1 9 


1 55 


1 ONOSOND 


DIFFRACT 




1 56 


1 ONOSP HE 


SOLUT 1 ON 




1 57 


1 RREGUL A 


SPHEROID 




6 1 9 


SPREAD 


1 NF 1 N 1 T 




655 


ALIGN 


SCALAR 




858 


ECHO 


UNIQUE 








VECTOR 


620 


6 2 0 


SQUARE 


rigid 










62 1 


4 4 


COLL 1 S 1 0 


DIFFRACT 




6b 


CYCLOTRO 


ELECTROM 




1 2 1 


homogene 


SOLUTI ON 




1 3 1 


I NC 0 H E R E 


CONDUCT 




3 1 b 


TRANSPOr 


1 NF 1 N 1 T 




3 1 7 


TR ANSVER 


SCREEN 




394 


INERTIA 


PLANE 




4 1 3 


NEUTRAL 


SLIT 




4 7 9 


UNIFORM 


WAVE 




62 1 


STATIC 






9 76 


GAS 


APPARATU 








SEARCH 


622 


1 72 


MATHEMAT 






3 8 6 


FOUR 1 E R 


1 ONOSPHE 




622 


STEADY 


NONSE ASO 








TEMPORAL 


623 


53 1 


E M 1 S S 1 


V AR 1 AT 1 0 




6 23 


ST I MUL 


DAYTIME 




73 7 


MASER 


DIURNAL 








SEASON 


624 


624 


STREAM 


SUMMER 








WINTER 


625 


4 2 9 


PLASTIC 


CYCLE 




6 2 5 


STRESS 


DAILY 




825 


ADHE 


LAYER 




836 


8 ONO 


MONTH 








YEAR 


62& 


1 6 3 


L 1 G H T N 1 N 






29 0 


SUCCESS 1 


M 0 N 0 JT A8 




6 03 


RETURN 


SPURIOUS 




626 


STROKE 


WAVEFORM 




706 


FLASH 


SELECT 








TIMING 


627 


156 


1 0 0 S P H E 



99 



627 


324 


V AH 1 AT 1 0 


642 




362 


DAYTIME 




609 


SEASON 






b27 


SUMMER 






652 


winter 






6 79 


DAILY 






728 


LAYER 






797 


south 






9b9 


DAY 


6 4 3 


6 2 8 


4 6 6 


SUNK 1 S E 






5 44 


He I gh t 






628 


SUNSET 






8 48 


DATA 




6 2 9 


6 1 5 


SOURCE 






629 


SURVEY 


6 4 4 


6 3 0 


3 0 


B 1 D 1 R E C T 






32 


bistable 






630 


SWITCH 






b b 5 


8 1 N A K 






735 


LOGIC 






774 


RELAY 






869 


flip 






9 77 


Gat 


6 4 5 


6 3 \ 


1 72 


MATHEMAT 






6 3 1 


bymbol 






632 


system 




6 3 2 


24 


A U TO M A T 1 






1 36 


informat 






4 0 4 


MACHINE 






50 7 


CLOSED 


646 




63 1 


SYMBOL 






6 3 2 


system 






782 


SERVO 


6 4 7 




83 0 


AXES 






9 66 


8 1 T 




6 33 


1 98 


N 0 R M A L 1 Z 






46 2 


stagger 






633 


TABULA 






719 


IMAGE 


6 48 


6 3 4 


87 


electron 






262 


SECONDAR 






53 1 


E M 1 S S 1 






63 4 


TARGET 




63 5 


1 2 


an I SOTRO 


6 4 9 




70 


D 1 ELECTR 






1 21 


homogene 






3 46 


CLASSIC 






399 


1 SUTHOP 






b3 5 


TENSOR 






703 


Field 


650 




74 1 


media 


6 5 1 


6 3 6 


33 


block I NG 






6 1 0 


SELECT 






636 


timing 






86 4 


fast 


6 52 


637 


63 7 


TOROID 






845 


CORE 




6 3 8 


638 


TR 1 A NG 




6 3 9 


5 


ADVANTAg 






423 


pentode 






639 


TR 1 ODE 






6 42 


tunnel 






8 1 4 


VALVE 


653 




827 


ANOD 




875 


GR 1 D 


654 


6 4 0 


935 


STAB 


6 5 5 


6 4 0 


TRIPLE 




69 1 


DOUBL 




o 


6 4 1 


TROUGH 




ERIC 


728 


LAYER 















656 


1 6 0 


junction 


CONVERT 




6 58 


ALLOY 


TR 1 ODE 








TUNNEL 


657 


4 06 


MOLECUL 


DIODE 




657 


AMMON 


GAIN 




737 


MASER 


stab 




BBB 


line 


1 DL 










658 


75 


D I RECT 1 V 


ASYMPTOT 




658 


AN AL Y 


SOLUTI ON 




88 3 


JUMP 


8 0 U N D A R 








FOURIER 


6 59 


6 59 


ANGLE 


FINITE 




8 1 8 


WEDGE 


SCALAR 








UNIQUE 


660 


4 9 0 


AERIAL 






660 


ARRAY 


ELECTRON 








EXC 1 TAT 1 


66 1 


1 45 


1 NTERFER 


IMPURITY 




200 


OBSER vat 


CONDUCT 




295 


telescop 


V ALE NC 




4 9 0 


AERIAL 


ATOM 




523 


D 1 AMET 


BAND 




573 


NE8ULA 






66 1 


ASTRO 


EXTR AORD 




77 1 


RADIO 


1 ONOGR AM 




993 


SUN 


VIRTUAL 








DENS 1 T 


662 


5 1 6 


C YL 1 ND 


RECORD 




66 2 


AXIAL 


valley 








CUR V 


663 


20 1 


OPERATIC 


PEAK 




3 1 4 


TRANS 1 ST 






520 


D E S 1 G N 


VAPOUR 




663 


BASIC 


WATER 




66 5 


8 1 N AR 






6 8 7 


DIGIT 


CIRCULAR 




735 


LOGIC 


MATHEMAT 








PERPEND 1 


664 


6 6 4 


BASIS 


FINITE 








scalar 


6 6 5 


258 


RE VERS 1 8 


VECTOR 




.36 4 


DECIMAL 


FIELD 




4 4 1 


READING 






630 


SWITCH 


CRYSTAL 




6 6 3 


BASIC 


E XC 1 TON 




66 5 


8 1 A R 


V 1 brat 




687 


DIGIT 


SHEAR 




735 


LOGIC 


MODE 




962 


ADD 






977 


GAT 


OBSER V AT 








PHOTOGR A 


666 


6 6 5 


BLOCK 


SHOWER 








VISUAL 


667 


205 


OUTBURST 


ECHO 




66 7 


BURST 






698 


ERUPT 


VOLUME 




7 0 5 


FLARE 






86 8 


F 1 NE 


EXOSPHER 








WH 1 STL 


668 


255 


RESONATO 


DUCT 




38 1 


EXCITED 






668 


C A V 1 T 


1 ONOSPHE 








VAR 1 AT 1 0 


66 9 


2 0 8 


PARTI CLE 


DAYTIME 




246 


REL AT 1 V 1 


season 




3 09 


TR A JECTO 


SUMMER 




66 9 


CH ARG 


WINTER 




7 9 8 


SPACE 


CYCLE 




896 


MASS 


daily 








LAYER 

DAY 


6 7 0 


6 7 0 


C 1 RCL 




6 7 1 


3 1 4 


TR ANS 1 ST 


A8 S 0 R 




370 


DISTORT 






37 7 


emitter 


ADAPT 




67 1 


CLASS 






9 1 4 


PULL 


1 RREGULA 








ASPECT 


6 72 


2 1 


ATMOSPHE 


SPREAD 




163 


L / GHTN 1 N 


ALIGN 




44 7 


RELEASE 


DUCT 




6 72 


CLOUD 



357 

6 39 

6 42 

6 89 

8 74 

935 

978 

20 

2 7 1 

339 

386 

538 

6 06 

643 

87 

94 

1 30 

350 

6 4 4 

829 

83 1 

1 00 

1 54 

48 1 

5 1 9 

59 7 

645 

8 46 

9 0 7 

6 46 

8 1 7 

40 

1 72 

2 1 2 

538 

6 05 

6 4 7 

703 

36 0 

382 

6 4 8 

785 

898 

200 

2 1 7 

6 1 2 

649 

858 

6 50 

95 

6 5 1 

856 

1 56 

324 

362 

609 

627 

652 

67B 

6 7 9 

728 

969 

653 

6 54 

1 57 

4 9 4 

6 1 9 

655 

856 



100 



673 


55 


CONF 1 GUR 


6B9 




3 39 


BOUNDAR 






b 1 6 


C YL 1 ND 






673 


CONIC 






696 


EQUAT 






772 


RATIO 


690 




840 


COAX 




6 7 4 


6 7 4 


CONNE 


69 1 


6 7 5 


2 1 8 


PHOTOSEN 






258 


RE VER S 1 B 






675 


COUNT 






76b 


PULSE 






962 


ADO 


6 9 2 


6 7 6 


57 2 


NARROW 






676 


OOUPL 






69 1 


DOUBL 


693 


6 7 7 


453 


SECTION 






677 


CROSS 




678 


2 


ACTIVITY 






6 


AFTERNOO 






4 6 7 


SUNSPOT 






564 


MED 1 AN 






609 


SEASON 






652 


WINTER 






6 7 8 


CYCLE 






7 4 9 


month 






960 


YEAR 






9 69 


DAY 


6 9 4 


6 79 


1 1 3 


geograph 






1 56 


1 ONOSPHe 






1 62 


latitude 






1 96 


NONSE ASO 






3 2 4 


VAR 1 AT 1 0 


6 9 5 




372 


DIURNAL 






564 


MEDIAN 






6 09 


SEASON 


6^6 




627 


SUMMER 






652 


winter 






6 79 


DAILY 






728 


LAYER 




68 0 


1 1 0 


FUNO AMEN 






2 03 


osci llat 






68 0 


DAMP 1 


69 7 


68 1 


4 1 4 


neutron 






68 1 


DECAY 


698 


682 


682 


DEF 1 N 




683 


6 1 0 


SELECT 






683 


DELAY 




684 


68 4 


DENSE 


699 


685 


209 


penetrat 






685 


DEPTH 






9 03 


OX 1 D 






93 0 


SKIN 




686 


6 8 6 


DER 1 V 


700 




7 46 


MOBIL 


6 6 7 


1 36 


1 NFORMAT 






364 


decimal 






404 


MACH 1 N E 






5 1 1 


C 0 Mp U T 






663 


basic 






fa 6 5 


B 1 N A R 






6 8 7 


D < 6 1 T 






8 54 


DRUM 


7 0 1 




9 4 0 


TAPE 




962 


add 


702 




96 6 


B 1 T 




968 


COD 




688 


688 


D 1 LUT 


7 03 




B 1 STABLE 


703 


\ 2 1 


homogene 


SUPERRE6 




169 


M AGNETOR 


TUNNEL 




28 7 


STRENGTH 


DIODE 




3 1 6 


TRANSPOR 






3 1 7 


TR ANSVER 


D 1 POL 




3 4 6 


CLASSIC 


1 NOUC 




56 9 


MOMENT 






6 3 5 


TENSOR 


STAGGER 




6 4 7 


VECTOR 


DESIGN 




7 03 


FIELD 


TRIPLE 




7 07 


FLUID 


OOUPL 




7 08 


FORCE 


D0U8L 




7 1 1 


GAUSS 


R E V E R S A 


704 


29 


8 ANDW 1 DT 


\/ELOC 1 T 




4 1 9 


P ARAMET 


DRIFT 




4 9 1 


AMPL 1 F 






7 0 4 


F 1 GUR 


ART 1 F 1 0 1 




87 4 


GAIN 


EXOSPHER 




9 00 


NO 1 S 


1 N T E R P L A 




9 1 5 


PUMP 


SATELL 1 T 




9 78 


1 DL 


terrestr 




9 8 2 


LOW 


orbital 








GR A V 1 T 


705 


2 05 


OUTBURST 


earth 




3 58 


CORPUSC 


ORBIT 




5 1 5 


COSMIC 


OUTER 




6 6 7 


BURST 


RADII 




6 9 8 


ERUPT 


BELT 




7 0 5 


FLARE 


MAIN 




736 


MAJOR 






7 9 4 


SOLAR 


0 1 RCULAR 




8 03 


STORM 


ext 1 not 1 




9 8 7 


RAY 


polar 1 z 








ELL 1 P 


7 06 


1 63 


L 1 GHTN 1 N 


PLANE 




4 73 


THUNDER 






626 


STROKE 


R E 0 1 P R 0 0 




7 06 


FLASH 


equal 










707 


1 2 1 


HOMOGENE 


1 NTEGR AL 




1 2 5 


HYDROMAG 


MATHEMAT 




1 6 6 


MAGNETIC 


SOLUTION 




3 5 0 


CONDUCT 


ST AT 1 ONA 




5 2 9 


DYNAMO 


CONIC 




58 7 


PLASMA 


EQUAT 




7 03 


FIELD 


E X P A N 




7 0 7 


FLUID 


SOLVE 










708 


1 6 6 


MAGNET 1 0 


ERROR 




2 08 


PARTICLE 






2 46 


RELAT 1 V 1 


OUTBURST 




4 0 1 


KINETIC 


PASSAGE 




7 03 


FIELD 


CORONA 




7 08 


FORCE 


BURST 




7 1 7 


HEL 1 0 


ERUPT 




870 


FLOW 


flare 










709 


1 1 1 


GALACTIC 


A SYMPTOT 




23 5 


RAD 1 AT 1 0 


1 MPERFEC 




573 


NE8 UL A 


SOLUTION 




7 09 


GALAX 


CONDUCT 




7 7 1 


RADIO 


1 NF 1 N 1 T 








EXACT 


7 1 0 


8 7 


ELECTRON 






1 9 1 


nitrogen 


APPROX 1 M 




2 2 7 


PRESSURE 


asymptot 




580 


OXYGEN 


diffract 




7 1 0 


GASES 


integral 




726 


1 ON 1 Z 


SOLUTION 




976 


GAS 


spheroid 




9 7 9 


1 0 N 


EQUAT 








expan 


7 1 1 


1 66 


MAGNETIC 


WEDGE 




703 


FIELD 






7 1 1 


GAUSS 


FAULT 










7 1 2 


1 1 4 


GEOMAGNE 


R ESONANC 




37 1 


DISTURB 


FERR 1 




59 1 


P ULS AT 


SPIN 




59 7 


RECORD 






7 1 2 


GIANT 


A N I S OTR 0 




9 76 


GAS 


0 OR R UG AT 









292 

6 4 2 

689 

690 

722 

4 6 2 

52 0 

6 4 0 

676 

69 1 

4 4 8 

480 

6 9 2 

1 7 

9 5 

1 49 

260 

3 00 

4 1 6 

5 4 0 

693 

7 55 

757 

7 7 0 

83 4 

8 9 4 

40 

99 

43 0 

694 

7 59 

239 

695 

1 4 1 

1 72 

.2 7 1 

2 8 4 

6 7 3 

696 

7 00 

7 96 

6 9 7 

2 0 5 

42 1 

5 1 4 

6 6 7 

6 98 

705 

20 

I 29 

27 1 

3 50 

395 

699 

1 5 

20 

72 

1 4 1 

27 1 

278 

696 

7 00 

8 1 8 

701 

2 54 

7 02 

93 4 

1 2 

63 



101 



7 t 3 


7 1 3 


GLASS 


728 




879 


H 1 GH 




7 1 4 


7 1 4 


GRAPH 




7 1 6 


7 1 5 


GROUP 




7 1 6 


7 1 6 


heavy 


729 


7 1 7 


708 


FORCE 


7 3 0 




7 1 7 


HEL 1 C 






88 8 


L 1 NE 


4 


7 1 8 


4 1 7 


0 VERSHO 


1 3 1 




7 1 8 


IDEAL 






7 1 9 


Image 


7^2 


7 1 9 


1 28 


1 MPEDANC 


7 3 3 




1 3 B 


1 NSERT 1 0 






23 3 


UU ADR 1 PO 






4 1 9 


P ARAMET 






49 8 


branch 






556 


LADDER 






633 


tabula 


7 3 4 




7 1 8 


IDEAL 






7 1 9 


IMAGE 






7 3 8 


MATCH 






86 7 


F 1 LT 


7 3 5 




905 


PASS 




720 


72 0 


1 MPUR 






74 3 


metal 






9 1 6 


PURE 




72 1 


2 36 


KAd 1 OSON 






445 


REFRACT 






72 1 


INDEX 


7 36 


722 


36 1 


CURRENT 






56 9 


MOMENT 






69 0 


D 1 POL 






72 2 


1 NDUC 


7 3 7 


723 


4 1 4 


NEUTRON 






59 0 


proton 






72 3 


INNER 






757 


OUTER 




724 


293 


SYNCHRON 






3 1 3 


TR ANS 1 EN 






35 7 


CONVERT 






579 


OUTPUT 


, 




724 


INPUT 






8 3 8 


CHOP 


7 3 8 




852 


DR 1 V 






86 5 


feed 






96 8 


COD 




725 


44 


CO LL 1 S 1 0 






1 2 3 


hydrodyn 






4 06 


magneto 






725 


Ionic 






726 


1 on 1 z 


7 3 9 


726 


1 2 3 


hydrodyn 






3 1 6 


TR ANSPOR 






406 


magneto 






4 1 3 


neutral 






420 


partial 






524 


D 1 FFUS 






7 1 0 


gases 






72 5 


Ionic 


740 




726 


1 ON I Z 






9 76 


gas 




727 


72 7 


joint 


74 1 


728 


6 


afternoo 






3 1 


B 1 FURC A T 






1 96 


NONSEASo 






325 


vertical 






3 7 9 


equator 




o 


466 


SUNR I SE 




ERIC 


6 09 


season 













SUMMER 


742 


43 


COINCIDE 


TROUGH 




65 


CRYOTRON 


WINTER 




1 36 


1 NFORMAT 


DAILY 




484 


ACCESS 


LAYER 




742 


M EMOR 


HOUR 




93 8 


S TOR 






966 


B 1 T 


LEADS 










743 


9 2 


E VAPOR AT 


AD 1 AB AT 1 




2 1 6 


PHOTOELE 


LEVEL 




26 2 


SECOND AR 


MASER 




3 9 7 


1 NSUL A T 






7 2 0 


1 MPUR 


light 




743 


METAL 






J 1 6 


WAFER 


LIMIT 




8 2 5 


ADHE 






9 0 3 


OX 1 D 


GEOMAGNE 




93 0 


SKIN 


midnight 




941 


THIN 


DISTURB 




948 


V ACU 


DIURNAL 








LOCAL 


7 44 


32 8 


W AVELENG 






52 3 


D 1 AMET 


OETONAT 




7 44 


METRE 


MIRROR 








L 0 C A T 


745 


7 45 


MINIM 


M 1 CR OM 1 N 


7 46 


84 


ELECTRIC 


OPERATIC 




3 1 6 


TRANSPOR 


ACCESS 




6 8 6 


D ER 1 V 


SWITCH 




7 4 6 


MOBIL 


BASIC 








B 1 N AR 


747 


747 


MODEL 


LOGIC 




9 3 9 


TAIL 


B 1 T 










748 


1 75 


M 1 C R 0 M 1 N 


OBSER V AT 




748 


MODUL 


CORPUSC 








FLARE 


749 


56 4 


MEDIAN 


MAJOR 




6 0 9 


SEASON 


EAST 




6 7 8 


CYCLE 






749 


MONTH 


AD 1 ABAT 1 




8 9 7 


MEAN 


POPULAT 1 








QUADRUPO 


7 5 0 


356 


CONTROL 


ACTION 




750 


MOTOR 


ST 1 MUL 




852 


DR 1 V 


AMMON 








LEVEL 


7 5 1 


362 


DAYTIME 


MASER 




46 6 


SUNR 1 SE 


SOLID 




544 


HEIGHT 


PUMP 




75 1 


NIGHT 






7 8 0 


SCALE 


1 MPEDANC 




86 3 


FALL 


TERM 1 NAT 








DESIGN 


752 


1 2 0 


HEM 1 SPHE 


LADDER 




1 5 6 


1 ONO SPHE 


IMAGE 




46 3 


STATION 


MATCH 




752 


NORTH 


F 1 LT 




762 


POLAR 


PASS 




797 


SOUTH 






96 5 


BAY 


ADM 1 TTAN 








MULT 1 POL 


7 5 3 


2 1 8 


PHOTOSEN 


REPRESEN 




753 


OPTIC 


CANONIC 








NETWORK 


754 


29 


B ANDW 1 DT 


LINEAR 




754 


OPT 1 M 


PASSI V 








M A TR 1 


755 


1 7 


ART 1 F 1 C 1 






8 1 


ECCENTR 1 


CR 1 T 1 CAl 




26 0 


S ATELL 1 T 


midnight 




4 1 6 


0R8 1 TAL 


MAX I M 




4 24 


PERIGEE 






46 0 


SPUTNIK 


A N 1 S DTR 0 




550 


1 NCL 1 N 


D 1 ELECTR 




693 


EARTH 


ELECTROM 




755 


ORBIT 


PROP AGAt 








MEDIUM 


7 56 


756 


ORDER 


TENSOR 








MEDIA 


7 5 7 


208 


particle 


WAVE 




590 


PROTON 






693 


EARTH 



6 2 7 

6 4 1 

652 

6 79 

7 28 

8 82 

729 

3 

7 3 0 

7 3 7 

7 3 1 

7 3 2 

1 1 4 

1 78 

3 7 1 

372 

733 

366 

56 8 

7 3 4 

1 75 

2 0 1 

4 8 4 

6 30 

6 6 3 

665 

735 

9 66 

2 00 

3 58 

7 05 

736 

8 5 7 

3 

222 

2 34 

4 86 

6 2 3 

6 57 

7 30 

7 3 7 

795 

9 1 5 

1 28 

299 

5 20 

5 56 

7 1 9 

7 38 

8 6 7 

9 05 

4 

1 88 

2 48 

3 40 

4 1 2 

5 59 

58 1 

7 39 

64 

1 78 

7 40 

1 2 

70 

86 

2 3 1 

5 6 5 

6 35 

741 

9S4 



102 



757 


723 


INNER 


769 




757 


OUTER 






834 


8ELT 


770 


7 58 


758 


PAPER 




759 


7 2 


DIFFRACT 






73 


0 1 MENS 1 0 


7 7 1 




212 


PERPEND 1 






277 


SPHER 1 C A 






339 


8 0 UND AR 






395 


1 NF 1 N 1 T 






51 3 


CONVEX 






607 


SCREEN 






694 


ELL 1 P 






759 


Plane 






8 1 8 


WEDGE 






93 1 


SLIT 




7 6 0 


760 


PLATE 


772 




785 


SHEAR 




76 1 


353 


CONTACT 


773 




76 1 


Point 






950 


VIEW 


774 


7 6 2 


1 1 4 


geomagne 






162 


LAT 1 TUDE 






37 1 


disturb 






752 


NDRTH 


7 75 




762 


POLAR 






803 


STORM 






996 


ZON 


7 76 


7 6 3 


83 


EFF 1 c 1 EN 






446 


REGUL AT 






468 


SUPPL 1 E 






599 


RECT 1 F 


7 77 




600 


RESOLV 






763 


Power 






792 


S I L 1 C 


778 




990 


ROW 






998 


DC 


779 


7 6 4 


429 


PLASTIC 


780 




764 


PRINT 






825 


ADHE 






836 


BOND 




7 6 5 


76 


D 1 SCH AR G 






87 


electron 






1 24 


hydrogen 


78 1 




227 


PRESSURE 






387 


GASEOUS 






545 


helium 






563 


ME ASUR 






587 


Plasma 






765 


PRDBE 


782 




976 


GAS 






97 9 


1 ON 




7 6 6 


1 80 


M 1 LL 1 U 1 C 






326 


WAVEFORM 






456 


shaping 






477 


TRIGGER 


783 




6 1 0 


SELECT 






675 


COUNT 


784 




766 


Pulse 






81 1 


TRAIN 


785 




922 


R 1 SE 


767 


767 


quant 






807 


THEOR 




7 68 


162 


latitude 






200 


OBSERV At 


786 




372 


DIURNAL 




379 


equator 






529 


dynamo 






768 


quiet 






794 


SOLAR 




Q ^ 9 


969 


DAY 


7 8 7 


769 


radar 




ERIC 

















ECHO 


788 


373 


DOPPLER 






788 


SHIFT 


RADIAL 




906 


PH AS 


EARTH 








RADII 


789 


CD 

>0 


SHOCK 


RING 










790 


790 


SHORT 


GAL ACT 1 C 








1 NTERFER 


79 1 


34 1 


CAP AC 1 T 


TELESCOP 




377 


EMITTER 


ANGULAR 




791 


SHUNT 


JUP 1 TER 




889 


LOAD 


BRIGHT 








NEBULA 


792 


1 1 6 


GERM AN 1 U 


ASTRO 




1 60 


JUNCTION 


GALAX 




436 


PROTECT 


RADIO 




449 


REVERSE 


S T A r. 




599 


RECT 1 F 






763 


POWER 


CONIC 




792 


S 1 L 1 C 


RATIO 










793 


6 1 0 


SELECT 


REDUC 




793 


S 1 NG L 


ADVANTAG 


79 4 


205 


outburst 


SWITCH 




467 


SUNSPOT 


RELAY 




51 4 


CORONA 


DUAL 




530 


ECL 1 PS 






7 05 


FLARE 


ASYMPTOT 




768 


QUIET 


ECCENTR 1 




794 


SOLAR 


RE VOL 




96 0 


YEAR 






993 


SUN 


SCATTER 








SCALAR 


795 


48 6 


ACTION 


RIGID 




737 


MASER 






795 


SOLID 


FARADAY 




934 


SPIN 


ROT AT 










796 


8 


ALTERNAT 


ROUND 




1 4 1 


1 NTEGRAL 






27 1 


SOLUTION 


S AMPL 




346 


CLASSIC 






538 


FINITE 


1 SOTHERM 




696 


EQU A T 


TUR8ULEN 




796 


SOLVE 


vertical 




967 


BOD 


DENS 1 T 








NIGHT 


797 


1 1 3 


GEOGRAPH 


SCALE 




1 t 4 


GEOM AGNE 






1 20 


HEM 1 SPHE 


CHARACTE 




1 56 


1 0 NO SPHE 


REVERSE 




200 


OBSER vat 


RECT 1 F 




372 


DIURNAL 


SELEN 




463 


STATION 


volt 




627 


SUMMER 






752 


NDRTH 


FEEDBACK 




797 


SOUTH 


RESPONSE 








CLOSED 


798 


1 49 


1 NTERPL A 


SYSTEM 




669 


CH ARG 


SERVO 




798 


SPACE 


LOOP 




8 1 5 


VEH 1 C 


SHAPE 


7 9 9 


76 


D 1 SCH ARG 






799 


SPARK 


SHARP 










8 0 0 


1 36 


1 N F 0 R M A T 


TH 1 CKNES 




589 


PRESET 


CRYSTAL 




800 


SPEED 


V 1 BRAT 




879 


HIGH 


PLATE 




9 1 9 


READ 


SHEAR 




938 


ST OR 


MODE 




940 


TAPE 






962 


ADD 


CORRUG AT 








PERPEND 1 


80 1 


1 52 


1 NVERS 10 


CONDUCT 




3 1 2 


transfor 


1 NF 1 N 1 T 




80 1 


SPLIT 


C YL 1 ND 








SHEET 


802 


802 


START 


shell 


803 


1 1 4 


GEOM AG NE 






1 84 


MORPHOL 0 



8 58 

592 

693 

7 70 

9 2 1 

1 1 1 

t 45 

295 

33 1 

4 00 

500 

573 

66 1 

7 09 

77 1 

936 

6 7 3 

772 

773 

5 

6 3 0 

7 7 4 

855 

20 

8 1 

7 7 5 

4 52 

6 06 

7 76 

384 

777 

7 78 

7 7 9 

1 59 

3 1 9 

325 

5 1 9 

751 

780 

3 9 

449 

599 

7 8 1 

951 

1 0 1 

257 

507 

632 

7 82 

892 

783 

784 

3 05 

36 0 

6 4 8 

760 

785 

8 98 

63 

2 1 2 

350 

395 

51 6 

786 

787 



103 



803 


205 


Outburst 




285 


S T AT 1 S T 1 




300 


TERRESTK 




5 9 1 


P U L S A i‘ 




705 


FLARE 




762 


POLAR 




803 


STORM 




894 


MAIN 




965 


BAY 


8 04 


804 


STRIP 




93 1 


SLIT 


8 05 


48 0 


VELOC 1 T 




805 


SURGE 


8 06 


1 08 


F KEuUE NC 




806 


SWEEP 


807 


88 


ELECTROS 




1 03 


FERRO MAC 




286 


S TOCH AS T 




346 


CLASSIC 




38 2 


EXC 1 TON 




401 


KINETIC 




529 


DYNAMO 




6 1 7 


SPHERE 




767 


UU ANT 




807 


THEOR 


8 08 


1 00 


EXTR AORD 




808 


TRACE 


8 09 


809 


TRACK 


8 1 0 


3 t 9 


TURBULEN 




438 


RAD 1 ant 




56 7 


METEOR 




6 1 2 


shower 




8 1 0 


TRAIL 




858 


ECHO 


8 1 1 


1 50 


INTERVAL 




766 


pulse 




81 1 


TRAIN 


8 1 2 


81 2 


TWIST 




888 


L 1 NE 




89 8 


MODE 


8 1 3 


2 1 


ATMOSPHE 




51 9 


DENS I T 




5 4 4 


HEIGHT 




8 1 3 


UPPER 


8 1 4 


423 


PENTODE 




488 


ADJUST 




49 7 


B A L A NC 




543 


heater 




639 


T R I ODE 




8 1 4 


VALVE 




827 


ANOD 




875 


GR 1 D 




995 


TUN 




99 8 


DC 


8 1 5 


26 0 


SATELL 1 T 




798 


SPACE 




8 1 5 


VEH 1 C 



8 1 8 


7 59 


PLANE 




8 1 8 


WEDGE 




8 59 


EDGE 


8 1 9 


8 1 9 


WEIGH 


82 0 


820 


W H 1 T E 




900 


NO 1 S 


82 1 


8 2 1 


WIDTH 


82 2 


1 1 3 


GEOGRAPH 




1 62 


LAT 1 TUDE 




463 


STATION 




8 22 


WORLD 


823 


65 


CRYOTRON 




1 3 4 


1 ND UCT AN 




2 52 


RESISTOR 




823 


WOUND 




8 4 1 


COIL 




957 


WIRE 


8 2 4 


8 24 


YIELD 


825 


50 


COMPONEN 




4 29 


PLASTIC 




6 25 


STRESS 




7 43 


METAL 




764 


PRINT 




225 


ADHE 




836 


BOND 




950 


VIEW 


826 


1 06 


FOR MAT 1 0 




3 35 


BALLOON 




826 


AL T 1 




9S " 


X 


82 7 


4 23 


PENTODE 




5 03 


CA“ HOD 




639 


TR 1 ODE 




8 1 4 


VALVE 




827 


ANOD 




865 


FEED 




8 75 


GR 1 D 


828 


1 49 


1 N T E R P L A 




5 00 


BRIGHT 




828 


AREA 




8 57 


EAST 




993 


SUN 


82 9 


58 


CONST 1 TU 




87 


ELECTRON 




495 


ATOMIC 




6 4 4 


VAL E NC 




8 2 9 


ATOM 




9 09 


PHOT 


83 0 


1 2 


AN 1 SOTRO 




85 


electrod 




88 


electros 




3 46 


CLASSIC 




5 1 6 


C Y L 1 ND 




6 1 7 


SPHERE 




632 


system 




830 


AXES 



8 16 3 9 

50 
743 
8 1 6 

817 646 

81 7 



CHAR ACTE 
COMPONEN 

.metal 

WAFER 

\/ A P 0 UR 
WATER 



8 1 8 




1 

72 
86 
1 4 1 
65 9 
70 0 



ACOUSTIC 
DIFFRACT 
ELECTROU 
I NTEGR AL 

angle 

EXPAN 



83 1 23 

29 
1 38 
462 
471 
4 78 
572 
6 4 4 
83 1 
867 
9 05 



ATT ENUAt 
band W I DT 
1 NSERT I 0 
stagger 

T E L E V I S 

tunable 

narrow 

V A L E NC 
BAND 
F I L T 
PASS 



832 158 IRREVERS 

454 SEGMENT 
832 BASE 



8 3 3 


289 

833 


8 3 4 


208 

235 

4 1 4 

5 I 5 
5 9 0 
693 
757 
B3 4 
9 4 4 
987 


83 5 


1 83 
3 1 4 
377 
8 3 5 
875 


836 


429 
6 1 4 
625 
76 4 
825 
836 
950 


83 7 


83 7 


838 


3 1 4 
49 1 
724 
838 
8 7 4 
99 8 


839 


839 
9 03 


6 4 0 


6 73 
8 4 0 
888 


8 4 1 


1 3 4 
823 
8 4 1 


842 


503 
8 4 2 


8 43 


843 


8 4 4 


329 
599 
8 4 4 


8 4 5 


385 
4 4 0 
4 5 0 
4 8 3 
63 7 
8 4 5 


8 4 6 


6 4 5 
8 4 6 
8 8 0 


8 4 7 


8 4 7 
93 9 


848 


23 6 
6 2 8 
8 4 8 


8 4 9 


244 
388 
849 
89 0 


85 0 


2 1 
26 0 
4 1 6 



SUBM I LL I 
BEAM 

PARTICLE 

R AD I AT I 0 

NEUTRON 

COSMIC 

PROTON 

EARTH 

OUTER 

BELT 

TRAP 

RAY 

MONOSTAB 
T R ANS I ST 
EMITTER 
BIAS 
GRID 

PLASTIC 

SILVER 

STRESS 

PRINT 

ADHE 

BONO 

VIEW 

CELL 

T R ANS I ST 

AMPL I F 

INPUT 

CHOP 

GAIN 

DC 

COAT 
OX I D 

CONIC 

COAX 

LINE 

I NDUCT AN 

WOUND 

COIL 

C A THOO 
COLD 

CONE 

AMBIENT 
R E C T I F 
COOL 

FERRITE 

REACTOR 

S AT UR AB 

WINDING 

TOROID 

CORE 

VALLEY 
CUR V 
HISS 

DARK 

TAIL 

R AD I OSON 

SUNSET 

DATA 

RE GENERA 
GENER AT 
D I V I 
LOCK 

ATMOSPHE 
S ATELL I T 
ORBITAL 



104 



850 


424 


PERIGEE 


8 6 5 




850 


DRAG 






963 


A 1 R 










86 6 


85 1 


8 5 1 


DRAW 




852 


1 0 1 


KEED8ACK 






3 4 5 


CIRCUIT 






4 4 6 


REGULAT 






552 


1 N VERT 






579 


OUTPUT 


86 7 




724 


INPUT 






7 50 


MOTOR 






852 


DR 1 V 






865 


FEED 






951 


VOLT 




853 


8 53 


DROP 




854 


2 4 


AUTOMAT 1 






1 66 


MAGNETIC 






4 0 4 


MACHINE 






4 4 1 


READING 


868 




4 84 


ACCESS 






5 1 1 


C 0 MPU T 






58 7 


DIGIT 






854 


DRUM 


869 




9 1 9 


READ 






938 


S TOR 






9 4 0 


TAPE 




855 


499 


8 R 1 DGE 






774 


RELAY 






855 


DUAL 




856 


23 1 


PROP AG AT 






5 5 1 


WH 1 STL 


8 7 0 




6 55 


ALIGN 






8 56 


DUCT 










87 1 


857 


1 1 5 


GEOPHYS 1 






7 36 


MAJOR 






828 


AREA 






8 5 7 


EAST 










872 


858 


1 57 


1 RREGULA 






2 69 


S 1 Ml‘LT an 


873 




2 72 


sound I NG 






2 8 0 


SPORAD 1 C 


8 7 4 




4 9 4 


ASPECT 






567 


METEOR 






6 1 9 


SPREAD 






6 49 


V 1 SUAL 






769 


RADAR 






8 1 0 


TRAIL 






8 58 


ECHO 




8 59 


8 1 8 


WEDGE 






859 


EDGE 




86 0 


53 1 


EM 1 SS 1 


8 75 




860 


emit 




8 fa 1 


86 1 


face 




862 


8 6 2 


fail 




863 


2 4 0 


F^ECOMB 1 N 






48 1 


virtual 






5 1 9 


DENS 1 T 






58 0 


OXYGEN 


8 7 6 




75 1 


N 1 GHT 






863 


fall 




864 


589 


PRESET 






6 1 0 


SELECT 


8 7 7 




636 


TIMING 






864 


F AST 




865 


59 


CONVENT 1 






1 4 7 


1 nternal 






724 


1 NPUT 






82 7 


anod 


8 78 




852 


D R ( V 




ERJC 

















FEED 

GRID 


8 79 


3 4 


8REMSSTR 






253 


RESOLUT 1 


CRYOTRON 




7 1 3 


GLASS 


EVAPORAT 




800 


SPEED 


DEPOSIT 




879 


HIGH 


C A R 8 0 N 
FILM 




986 


0 1 U 


THIN 


880 


1 8 


ASSOC 1 AT 






846 


CUR V 


ATTENUAT 




8 8 0 


HISS 


1 N S E R T 1 0 
NORM AL 1 Z 


b 8 1 


88 1 


HOLE 


TORSION 
8 R ANCH 


882 


324 


VAR I AT 1 0 


LADDER 




372 


DIURNAL 


IMAGE 




728 


LAYER 


MATCH 




882 


HOUR 


8 AND 
F 1 LT 




969 


DAY 


PASS 


883 


2 1 3 


PHENOMEN 






257 


RESPONSE 


STRUCTUR 




658 


AN AL Y 


8 UR S T 
FINE 




883 


JUMP 




884 


884 


LAND 


B 1 ST A8LE 
COMPLEME 
M0N0STA8 


885 


885 


LEAD 


MULT 1 V 1 8 
OPERATIC 


886 


8 8 6 


LEAK 


TRANS 1 ST 
TRIGGER 


887 


88 7 


LIFE 


SWITCH 


8 88 


465 


STR A 1 GH 


FLIP 




657 


AMMON 






7 1 7 


HEL 1 C 


FORCE 




8 1 2 


TWIST 


FLOW 




840 


COAX 






888 


LINE 


GALACTIC 
R AD 1 AT 1 0 


8 89 


79 1 


SHUNT 


COSMIC 




889 


LOAD 


FLUX 


89 0 


1 08 


FREQUENC 


FORM 




203 


05C 1 LL AT 






588 


PRECIS 


FREE 




589 


PRESET 






8 49 


D 1 V 1 


8 AND W 1 DT 




890 


LOCK 


CONVENT 1 
AMPL 1 F 




9 08 


PHAS 


TUNNEL 


89 1 


89 1 


LONG 


F 1 GUR 




939 


TAIL 


CHOP 

GAIN 


892 


1 0 1 


FEEDBACK 


PULL 




507 


CLOSED 


PUMP 




782 


SERVO 


STA8 

D8 




892 


LOOP 


PENTODE 


89 3 


23 


ATTENUAT 




1 3 8 


1 NSERT 1 0 


STAB! L 1 




893 


LOSS 


TR 1 ODE 
VALVE 




971 


EDD 


ANOD 


89 4 


1 1 4 


GE0MA6NE 


8 1 A S 




529 


0 YNAWO 


FEED 




693 


EARTH 


GRID 




8 03 


STORM 


Pull 




89 4 


MAIN 


COLL 1 S 1 0 

extr aord 




93 9 
9 9 6 


TAIL. 

ZON 


MAGNETO 

GYRO 


89 5 


89 5 


MASK 


magnet I C 

RECORD 


896 


669 

896 


CHARG 

MASS 


HEAD 


89 7 


749 


MONTH 


READ 

TAPE 




89 7 


MEAN 


WRIT 


898 


28 1 


SPURIOUS 


thermal 




6 4 8 
785 


V 1 8R AT 
SHEAR 


HEAT 




8 1 2 


TWIST 



865 

875 

65 

92 

365 

50 1 

866 

9 4 1 

23 

1 38 

1 98 

4 74 

4 98 

556 

7 1 9 

7 38 

83 1 

867 

9 05 

2 88 

6 6 7 

868 

32 

49 

1 83 

1 90 

2 0 1 

3 1 4 

477 

630 

869 

7 08 

870 

1 1 1 

235 

5 1 5 

871 

872 

8 73 

29 

59 

49 1 

6 42 

7 04 

838 

8 74 

9 1 4 

9 1 5 

9 35 

997 

4 23 

461 

639 

8 1 4 

827 

835 

865 

875 

9 1 4 

44 

1 00 

4 06 

8 76 

1 66 

59 7 

877 

9 1 9 

9 4 0 

9 59 

4 72 

878 



105 



BBS 


898 


MODE 


9 1 5 


B99 


899 


moon 






923 


ROCK 






983 


LU N 


9 1 6 


9 0 0 


26 


BACKGROU 






292 


S 1 j P E R R E G 






7 04 


F 1 GUR 


9 1 7 




820 


WHITE 






9 00 


NO 1 S 






9 1 5 


PUMP 


9 1 8 




92 7 


SHOT 


9 1 9 


9 0 1 


1 58 


1 RRE VERS 






9 0 1 


NOTE 




9 02 


9 02 


OPEN 




903 


92 


E V AP OR A T 






262 


SECO ND AR 






685 


DEPTH 






743 


METAL 






839 


COAT 






903 


OX 1 D 


9 2 0 


9 04 


239 


R EC 1 PR OC 






904 


PAIR 


9 2 1 


9 05 


1 9 


AS YMMETR 






1 38 


1 NSERT 1 0 






263 


SELECT 1 V 


92 2 




29 9 


TERM 1 NAT 






462 


STAGGER 






498 


BRANCH 






502 


C A SC A D 






556 


LADDER 






7 1 9 


IMAGE 






738 


MATCH 






83 1 


BAND 


9 2 3 




867 


F 1 LT 






905 


PASS 






982 


LOW 


924 


9 06 


906 


PATH 




907 


645 


VALLEY 


925 




907 


PEAK 




9 08 


265 


S E M 1 D 1 U R 






552 


1 NVERT 






7 88 


SHIFT 


926 




890 


LOCK 






908 


PH AS 


9 2 7 


9 09 


2 1 6 


phot dele 






829 


atom 




9 1 0 


909 


PHOT 


928 


25 7 


response 




9 1 0 


PLOT 


929 


9 1 1 


1 09 


function 


9 3 0 




3 1 1 


transfer 






9 1 1 


POLE 




9 1 2 


9 1 2 


PORT 




9 1 3 


9 t 3 


PROD 


9 3 1 




988 


RED 


9 1 4 


28 


balanced 






43 7 


(^u AL 1 T Y 






67 1 


CLASS 






874 


gain 






875 


Grid 






9 1 4 


PULL 




9 1 S 


29 


B A ND W 1 D T 


932 




222 


POPULAT 1 




4 1 9 


paramet 


933 




49 1 


AMPL I F 




704 


F 1 GUR 






737 


maser 




ERJC 















874 


GAIN 






9 00 


NO 1 S 


934 


1 03 


9 1 5 


PUMP 




167 


9 78 


1 DL 




1 93 








286 


720 


1 MPUR 




569 


9 1 6 


PURE 




7 02 








795 


478 


TUNABLE 




934 


9 1 7 


RANG 


935 


5 


9 1 8 


RATE 




3 1 4 
46 1 


44 1 


READING 




535 


5 1 1 


. DMPUT 




639 


8 00 


SPEED 




642 


854 


DRUM 




8 74 


877 


HEAD 




93 5 


9 1 9 


READ 






938 


STOR 


9 3 6 


26 1 


9 40 


TAPE 




77 1 


9 59 


WRIT 




936 


966 


e 1 T 


9 3 7 


257 


1 09 


FUNCTION 




3 1 3 


2 2 1 


POL YNOM 1 




4 1 7 


920 


REAL 




93 7 


7 70 


RADII 


9 3 8 


65 


9 2 1 


R 1 NG 




1 36 








404 


3 3 


BLOCKING 




48 4 


1 8 0 


M t LL 1 M 1 C 




742 


388 


GENER AT 




8 0 0 


4 1 7 


OVERSHO 




854 


766 


PULSE 




9 1 9 


9 22 


RISE 




93 8 


9 42 


TIME 




940 

966 


469 


SURFACE 






899 


MOON 


939 


529 


923 


ROCK 




747 








84 7 


296 


TEMPER AT 




39 1 


9 24 


ROOM 




894 

939 


1 3 


apparatu 






253 


RESOLUt I 


940 


1 36 


295 


TELESCOP 




364 


925 


SCAN 




44 1 








5 1 1 


2 03 


OSC 1 LL A T 




59 7 


926 
1 05 


self 




687 
8 00 


FLUCTUAT 




854 


9 00 


NO 1 S 




87 7 


9 27 


SHOT 




9 1 9 
93S 


9 28 


S 1 GN 




940 


929 


SIZE 


94 1 


74 3 


350 


conduct 




866 
94 1 


492 


ANOMAL 




685 


depth 


9 42 


1 5 0 


743 


metal 




4 1 7 


9 30 


SKIN 




443 


40 


C 1 RCULAR 




4 7 5 
9 2 2 


72 


D 1 F FR A C T 




9 4 2 


3 9 1 


1 NC 1 DEN 




98 0 


607 


SCREEN 




7 59 


Plane 


943 


1 0 8 


8 04 


STRIP 




29 3 
94 3 


93 1 


SLIT 




9 54 


WAVE 




9 72 


FAR 


944 


206 


9 32 


SLOW 




366 
59 0 


1 06 


F OR M A T 1 0 




83 4 
9 4 4 


9 3 3 


SOFT 




98 7 


987 


RAY 




9 99 


X 


945 


945 



FERRO MAG 
MAGNET I S 
NONM AGNE 
STOCHAST 
MOMENT 
FEHR I 
SOLID 
SPIN 

ADVANTAG 
TRANS I ST 
ST AB I L 1 
EXTREM 
T R 1 ODE 
TUNNEL 
GAIN 
STAB 

SC I NT 1 LL 

RADIO 

STAR 

RESPONSE 
TRANS I EN 

0 V ER SH 0 
STEP 

CRYOTRON 

f N F 0 R M A T 

MACHINE 

ACCESS 

MEMOR 

SPEED 

DRUM 

READ 

S TOR 

TAPE 

B I T 

DYNAMO 

MODEL 

DARK 

LONG 

MAIN 

TAIL 

1 NFORM AT 
DECIMAL 
READING 
COMPUT 
RECORD 
DIGIT 
SPEED 
DRUM 
HEAD 
READ 

S T 0 R 
TAPE 

METAL 

FILM 

THIN 

I NTERV AL 

OVERSHO 

RECOVER 

TRANSIT 

RISE 

TIME 

LAG 

frequenc 

S YNCHR ON 
TONE 

particle 

DETONAT 

PROTON 

BELT 

TRAP 

RAY 



TUBE 



10b 



946 


946 


UN 1 F 


947 


9 4 7 


UNIT 




96 2 


ADD 


94 8 


92 


EV APORAT 


743 


metal 




9 48 


VACU 


9 4 9 


9 49 


VARY 


9S0 


3 08 


toleranc 


429 


PLAST ^ C 




76 1 


POINT 




825 


ADHE 




336 


bond 




950 


VIEW 


95 1 


83 


EFF 1 C 1 EN 


307 


thyratro 




440 


REACTOR 




446 


REGULA T 




468 


SUPPL 1 E 




497 


B AL A NG 




S43 


HEATER 




/8 1 


SELEN 




8 52 


D n 1 V 




95 1 


VOLT 




986 


0 1 L. 




995 


TUN 




998 


DC 


9 52 


952 


WALL 


9 53 


953 


WASH 


954 


1 


ACOUSTIC 




27 


BACKWARD 




0 3 


CORRUGAT 




7 2 


U 1 FFRACT 




86 


ELECTROM 




99 


EXT 1 NCT 1 




1 79 


M 1 LL t MET 




231 


PROPAGAT 




283 


STANDING 




289 


S U B M 1 L L 1 




3 ! 8 


TRAVELL 1 




i>6 5 


MEDIUM 




b 0 7 


SCREEN 




74 1 


MEDIA 




93 1 


SLIT 




954 


WAVE 


955 


955 


WELL 


956 


2 1 


ATMOSPHE 




1 22 


HOR 1 ZO NT 




537 


FADING 




956 


W 1 ND 


9 57 


823 


WOUND 




957 


WIRE 


958 


958 


WORK 


959 


44 t 


READING 




877 


HEAD 




9 1 9 


READ 




9 59 


WRIT 


960 


2 


ACTIVITY 




1 1 5 


GECPHYS 1 




1 48 


I NTERN AT 




1 62 


LAT 1 TUDc 




3 24 


VAR f AT 1 0 




46 3 


ST AT ION 




467 


SUNSPOT 




564 


MEDIAN 




6 09 


SEASON 




678 


CYCLE 




794 


SOLAR 




9 60 


YE AR 



9C 1 


9 6 1 


ZERO 


962 


201 


OPERATIC 




36 4 


DECIMAL 




579 


OUTPUT 




6 65 


B 1 N AR 




6 75 


COUNT 




6 87 


DIGIT 




8 0 0 


SPEED 




947 


UNIT 




9 6 2 


ADD 


963 


2 t 


ATMOSPHE 




8 50 


DRAG 




963 


A 1 R 


964 


1 1 1 


GALACTIC 




5 00 


BRIGHT 




573 


NEBULA 




96 4 


ARC 




9 8 8 


RED 


965 


1 1 4 


GEOMAGNE 




1 78 


MIDNIGHT 




496 


AURORA 




59 1 


PULSAT 




7 52 


NORTH 




8 03 


STORM 




9 65 


BAY 


966 


3 64 


DECIMAL 




4 8 4 


ACCESS 




589 


PRESET 




632 


SYSTEM 




687 


DIGIT 




7 35 


LOGIC 




7 42 


MEMOR 




9 1 9 


READ 




9 38 


STOR 




966 


B 1 T 


967 


40 


C 1 R CUL AR 




7 2 


DIFFRACT 




1 2 1 


HOMOGENE 




278 


SPHEROID 




395 


1 NF 1 N 1 T 




5 1 3 


CONVEX 




594 


R A D 1 US 




7 96 


SOLVE 




967 


80D 


968 


364 


DECIMAL 




44 1 


READING 




6 87 


DIGIT 




7 24 


1 NPUT 




9 68 


COD 


96 9 


627 


SUMMER 




652 


W INTER 




6 7 8 


CYCLE 




7 68 


QUIET 




882 


HOUR 




969 


DAY 


9 7 0 


9 70 


D 1 S 




993 


SUN 


97 1 


36 1 


CURRENT 




893 


LOSS 




9 7 1 


EDD 


972 


93 1 


SLIT 




9 72 


FAR 


9 73 


252 


RESISTOR 




3 23 


VARIABLE 




3 4 1 


CAP AC 1 T 




973 


F 1 X 


9 74 


9 74 


FOC 


9 7 5 


975 


GAP 


9 76 


3 1 6 


transpor 



976 


4 1 3 


NEUTRAL 


6 2 1 


STATIC 




7 1 0 


GASES 




7 1 2 


GIANT 




7 26 


1 ON 1 Z 




765 


PROSE 




976 


GAS 


977 


3 0 


8 1 D 1 R E C, T 


5 8 9 


PRESET 




6 3 0 


SWITCH 




665 


B 1 NAR 




977 


GAT 


9 78 


1 08 


FREOUENc 


4 1 9 


P AR AMET 




4 9 1 


AMPL 1 F 




6 1 3 


SIGNAL 




642 


TUNNEL 




7 0 4 


F 1 GUR 




9 1 5 


PUMP 




978 


1 OL 




990 


ROW 


979 


52 


C 0 NC E N T R 




87 


ELECTRON 




1 9 1 


nitrogen 




4 1 3 


NEUTRAL 




524 


D 1 FFUS 




7 1 0 


GASES 




765 


PROBE 




979 


1 ON 


98 0 


9 42 


T 1 ME 




9 80 


LAG 


98 1 


472 


THERMAL 




9 8 1 


LAW 


9 8 2 


535 


E X T R E M 




7 0 4 


F 1 GUR 




9 05 


PASS 




982 


LOW 


983 


1 4 


APPARENT 


265 


S E M 1 D 1 U R 




3 25 


VERTICAL 




899 


MOON 




983 


L UN 




994 


T 1 U 


984 


1 08 


FREOUENC 




1 1 2 


GAUSS 1 AN 




6 1 3 


SIGNAL 




9 8 4 


M 1 X 


98 5 


8 4 


ELECTRIC 




9 8 5 


NET 


986 


8 7 9 


HIGH 




95 1 


VOLT 




986 


0 1 L 


987 


3 4 


8REW SST R 




14 9 


1 NTERPL A 




3 00 


TERRESTR 




335 


8 ALL 0 ON 




4 1 4 


neutron 




5 1 5 


COSMIC 




590 


PROTON 




705 


FLARE 




834 


8ELT 




933 


SOFT 




94 4 


TRAP 




9 87 


RAY 


986 


235 


R AD 1 AT 1 0 




9 1 3 


PROD 




9 64 


ARC 




988 


RED 


989 


989 


ROD 


99 0 


1 92 


NONL 1 NE A 



990 


2 3 7 


REACT ANC 


993 


993 


SUN 


996 


9 9 6 


ZON 




2 56 


RESTRICT 
















439 


REACT 1 V 


994 


1 8 4 


MORPHOLO 


997 


23 


ATTENU AT 




763 


POWER 




265 


SEM 1 D 1 UR 




874 


GAIN 




9 7 8 


1 DL 




324 


VAR 1 AT 1 0 




99 7 


D8 




9 9 0 


ROW 




529 


DYNAMO 
















983 


LUN 


9 98 


446 


REGUL A T 


99 1 


9 9 1 


SET 




994 


T 1 D 




468 


SUPPL 1 E 
















497 


B AL ANC 


992 


992 


SUM 


995 


446 


REGUL AT 




763 


POWER 










462 


STAGGER 




8 1 4 


VALVE 


993 


1 45 


1 NTERFER 




468 


SUPPL 1 E 




838 


CHOP 




235 


R A D 1 A T 1 0 




8 1 4 


VALVE 




951 


VOLT 




300 


TEHRESTR 




9 5 1 


VOLT 




995 


TUN 




500 


BRIGHT 




995 


TUN 




9 9 8 


D C 




5 J 4 


CORONA 




9 9 8 


DC 










6 6 1 


ASTRO 








999 


34 


8REMSSTR 




794 


SOLAR 


996 


496 


AURORA 




8 2 6 


ALT 1 




828 


AREA 




7 62 


POLAR 




933 


SOFT 




9 7 0 


0 1 S 




8 94 


MA I N 




999 


X 



o 




107 



6 

156 

196 

32U 

362 

372 

609 

627 

652 

678 

679 

728 

6 

1 56 

196 

32^4 

362 

372 

609 

627 

652 

676 

679 

960 

6 

176 

1 96 

372 

726 

6 

176 

372 

733 

6 

196 

32^4 

372 

564 

609 

652 

67S 

679 

728 

6 

196 

324 

372 

564 

609 

652 

678 

679 

960 

6 

1 96 

324 

372 

609 

627 

652 

679 

728 

882 



APPEr^DIX A6 



SELECriCHI raiM CLUSTER SET C3 , USED lU RUN l4(MCS0l) 





6 


AFTERNOO 




1 96 


NOl'JSEASO 




466 


SUNRISE 


Ah'TERNOO 


609 


SEASDl'J 


lONOSPHE 


679 


DAILY 


NONSEASQ 


72 a 


LAYER 


VARIATIO 






DAYTir® 


6 


AFTERNOO 


DIURNAL 


564 


INDIAN 


SEASON 


609 


SEASON 


SUMtCR 


652 


WINTER 


WINTER 


678 


CYCLE 


CYCLE 


679 


DAILY 


DAILY 


749 


MONTH 


LAYER 


960 


YEAR 


AFTERNOO 






lONOSPHE 






NONSEASO 






VARIATIO 


17 


ARTIFICI 


DAYTIME 


21 


ATMOS PKE 


DIURNAL 


260 


SATELLIT 


SEASON 


416 


ORBITAL 


SUMMER 


424 


PERIGEE 


WINTER 


693 


EARTH 


CYCLE 


769 


ORBIT 


DAILY 


850 


DRAG 


YEAR 








21 


ATMOS PHE 


AFTERNOO 


159 


ISOTHERM 


MIDNIGHT 






NONSEASO 


21 


ATMOS PHE 


DIURNAL 


163 


LIGHTNIN 


LAYER 


672 


CLOUD 


AFTERNOO 


21 


ATMOS PHE 


MIDNIGHT 


236 


RADIOSON 


DIURNAL 






LOCAL 


21 


ATMOS PHE 




416 


ORBITAL 


AFTERNOO 


850 


DRAG 


NONSEASO 


963 


AIR ■ 


VARIATIO 






DIURNAL 


21 


ATMOS PHE 


rCDlAN 


580 


OXYGEN 


SEASON 






WIr^^TER 


21 


ATMOS PHE 


CYCLE 


813 


UPPER 


DAILY 






LAYER 


21 


ATMOS PHE 




956 


WItCi 


AFTERNOO 






NONSEASO 




. . . 


VARIATIO 




. . i 


DIURNAL 






fCDlAN 


24 


AUTOMATI 


SEASON 


136 


INF0R41AT 


WI4>JTER 


363 


DECir^AL 


CYCLE 


441 


READING 


DAILY 


632 


SYSTEM 


YEAR 


687 


DIGIT 




854 


DRUM 


AFTL’RNOO 


966 


BIT 


NONSEASO 






VARIATIO 


24 


AUTOMATI 


DIURNAL 


136 


INFORMAT 


SEASON 


363 


DECimL 


sunr^R 


441 


READING 


WIr^^TER 


665 


BINAR 


DAILY 


687 


DIGIT 


LAYER 


854 


DRUM 


HOUR 


940 


TAPE 




966 


BIT 




968 


COD 



24 


AUTOr^TI 


44 


136 


INFORMAT 


100 


363 


DECIMAL 


406 


441 


READING 


876 


687 


DIGIT 




85 ^ 


DRUM 


44 


919 


READ 


121 


940 


TAPE 


246 


966 


BIT 


316 






401 


24 


AUTOr^TI 


621 


136 


INT’ORMAT 




404 


MACHINE 


44 


632 


SYSTEM 


121 


667 


DIGIT 


246 


854 


DRUM 


31 6 


966 


BIT 


401 


24 




7 O 8 


AUTOMATI 




136 


INT’ORMAT 


44 


404 


r^CHi;^ 


121 


6 S 7 


DIGIT 


246 


854 


DRUM 


316 


91 9 


READ 


413 


940 


TAPE 


621 


966 


BIT 


44 






121 




. . . 


316 


44 




401 


COLLISIO 


621 


37 


ELECTRON 


703 


316 


TRANS POR 




401 


Kir^TIC 


44 


621 


STATIC 


121 


44 




31 6 


COLLISIO 


401 


87 


ELECTRON 


703 


316 


TRANS POR 


708 


413 


r'lEUTRAL 




524 


DIFFUS 


44 


726 


I0NI2 


123 


976 


GAS 


316 






40 6 


44 


COLLISIO 


413 


87 


electron 


725 


316 


TRANS POR 


726 


413 


NEUTRAL 




621 


STATIC 


44 


726 


lONIZ 


123 


976 


GAS 


406 


44 




413 


COLLISIO 


587 


87 


electron 


725 


41 3 


neutral 


726 


524 


DIFFUS 




726 


lONIZ 


44 


976 


GAS 


123 


979 


ION 


406 


44 




725 


COLLISIO 


876 


87 


Electron 


413 


r^EUTRAL 


44 


587 


PLASr^ 


131 


621 


STATIC 


246 


726 


lONIZ 


41 3 


976 


GAS 


621 


44 


COLLISIO 


44 


87 


ELECTRON 


131 


41 3 


NEUTRAL 


41 3 


587 


PLASMA 


587 


726 


lONiZ 


621 


976 


GAS 


976 


979 


ION 



COLLISIO 

EXTRAORD 

r^AGNETO 

GYRO 

COLLISIO 

HC.10GENE 

RELATIVI 

TRAUSPQR 

KII'IETIC 

STATIC 

COLLISIO 

HOMOGEIsIE 

RELATIVI 

TRANS POR 

KirvIETIC 

FORCE 

COLLISIO 

HOI^OGEIvIE 

RELATIVI 

TRANS POR 

t^UTRAL 

STATIC 

COLLISIO 

HOMOGEI>IE 

TRANS POR 

KI44ETIC 

STATIC 

FIELD 

COLLISIO 

HOMOCEI>IE 

TRANS POR 

KINETIC 

FIELD 

FORCE 

COLLISIO 

HYDRODYN 

TRANS POR 

r4AGr>IET0 

r^UTRAL 

IONIC 

lONiZ 

COLLISIO 

HYDRODYN 

r^AGNETO 

r>4EUTRAL 

PLASr4A 

IONIC 

lONIZ 

COLLISIO 

HYDRODYN 

[4AGI^IET0 

IONIC 

GYRO 

COLLISIO 

INCOHERE 

RELATIVI 

NEUTRAL 

STATIC 

COLLISIO 

INCOHERE 

r^UTRAL 

PLASMA 

STATIC 

GAS 
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A P P E N n I X 



B I 



Cor^PLETE LIST OF HEqUESTS 



showing the name of the requestor, the number of key-word stems, the stems 
themselves with their urxjerlininf:B, if any, (section VI. 2 . 6 ), and their 
dictionary code numbers (Append ix A 1 ). 



o 

ERIC 



Remmst 0 (P*H* Hanunond) (6) 

i'’UV::'IJ.*:KCY OoCILLVrOitS TRi'JISISTOHS THEORETIC/iL TUE/ITI-UCIIT /JID PUi\CTIC/kL CIRCUIT DETAILS 

•:Tr,if KirCuEKC OSCILL^T mVISIST theor circuit 

- 345 



879 103 



203 



314 



007 



1 (Cancelled) (10) 

H;:GEn.-:iL.Tivr: ia;cKi‘/i-:ns iti tii: liiGH i-TiEgunNCY region v/ith details op noise piGUiiR sklectivjty /^kd radi.oted 

' Rr:GSNE)L\ KJi’CKiV HIGH FlLEgUENC NOTG FIGURE GEUICTIV 

2Vv 596 879 106 900 70L 263 593 

SIGL'AL LEViiL 
SIGIL'kL LEVEL 

613 730 

Henuest 2 (P«H, Ilunnond) (5) 

of dielectric coRsviJiT op li:-uids by the use of microwave technu^ues 

;7^\SUH DISU:CTO consivjl LIQUID MICR07/AV 

563 70 352 560 177 

Rc!'’'iest 3 (P*’i, (Ini;itnond) (61 

:AT.:E:.iAx'IC,.L /J.'ALYGIG -J/D design details op V/AVKGUIDK PiiD MICROV/AVB JL'VDIATIONS 
i.^.TilriVAT *:;.\LY DESIUi; './iV/EGUID NICROV.'AV il\di/.t 

172 650 5^0 327 177 593 

HQ^uest L (P.H. .Hammondl (9) 

: ;E of digital C0MH!T.;:;A it; TIIE design of B/OID pass filters ILWING given PILISE UiD ATTEinJTATlOK C'..mC?.*:;USTIC3 



DIGIT 

687 






DESIGN 

520 



DxOlD PASS FILT 

831 905 867 



H!AS 

9 O 8 



ioriEinjAT 

23 



CIL.RACTS 

39 



Request 5 (F,M, Blake) (5) 

SYSTI'tUS OF DATn CODING FOR ItJFOltiUTION TRANSFER 
SYS. ’EM Dip'Ll COD LIFORMIT TRxJi’GFBR 

632 848 968 136 311 

Request 6 (Cancelled) (3) 

US.-; OF PROGIUHsS HI KIIGIiaEHING TESTING OF CONPuTEllS 

raOG:oE.I Ji.GIHEER COMRJT 

435 89 512 

Request 7 (F,L, Blake) ( 5 ) 

SII.niL.TlA'ii OF :i\Tii3I.L;TICAL FDTtCTIONS USING MAGNETIC CIRCUITS 
SniULT ’.'ATIKj.jkT FUNCTION MAGNETIC CIRCUIT 

a 5Q 172 109 345 

Request 8 (Cancelled) (A) 

MULTIPLE DIGIT TECHNI ,m‘*S H; PONT DECIMAL ADDIiESS 

;.;IiLTIPL DjGIT decimal ADD 

w 687 363 962 

Request ’9 (F*M* Blake) (4) 

NUMBER REP. i-tSENTATION Ii: BDNJIY M/.CHINES ■ 
irUMIKR REPRESSN DUDiR MACHINE 

575 2i3 665 404 

Request 10 (i.'.L, Price) ( 7 ) 

SECOND;J^Y EMISSION OP ELECTRONS BY POSITIVE ION D0MBi\RDM3iNT OF TIE CATHODE 
SECQND/JI Ei.lISSi ELl-JCTltON POSITIVE ION D01.1B/\RD CATHOD 

2^“ 531 224 979 "333 503 



Request 11 . (.;,L, Price) 

MEiVSimniENT 0? PLT3I1T TEMPI-IR/kTUREG 
MryiSUR PIV,J3Mi\ TSMPSRAT 

563 587 296 



(7) 

IN /lRC discharge using shock VtAVE TECHNIQUES 

ARC DISCHARG SHOCK WAVE 

^ 76 ‘1ST 954 



Request 12 (\7,L, Price) (7) 

C:iiul;iC'i’ERISTICS OF TiE SINGLE ELECTRODE DISCHARGE IN THE RARE GASES AT LOW PRESSURES 
C:LMl\CTE SD^GL ELECTROD DISCHARG GASES LOW HIESSURE 

39 793 55 76 .710 982 227 

Request I3 (V/.L, IVioe) (5) 

METHODS OF CALCULvTING BsSTANTANEOUS POWI-CR DISSIPATION IN REACTIVE CIRCUITS 
CuLCULiT POVffiR DISSIPA^ REACTIV CIRCUIT 

35 763 79 439 3 45 

Request 14 (V/,L, Price) (3) 

Tits EFFECT OF OXIMTION ON CIRCUIT BREAKER CONTACTS 
OXID CIRCUIT CONTACT 

903 345 353 

Request I5 (V/, Fincham) (7) 

TEMPERivTURK Ii;Df:PK‘;DEi;T METHODS FOR TUNING HIGHLY STABLE HIGH FREQUENCY OSCILIATORS 
TEMP3R/iT L, DEPEND TUN (HIGH) STAB HIGH FREQUFJJC OSCILIAT 

296 133 995 (879) 935 " 5 t 9 ’" 1 oS 203 
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Request 1u {./. (8) 

,.:d g:uu'j mi riffl ix'/mou o? T.in:;o.iu'A<ui;:ir:D Tmir'.i) Pa:;:; ir.riG ...MriJnKHG 
gigu^ii dk:-:;ign TiLUiiJiGT tun p.\ 8 :; b..:;u -miltp 

1/2 714 i)2o 314 995 905 831 491 



Request 1? (V/. K ioicha n) (6) 

.;r:.v ... r.G op pirv.v.jj iLWiiiG .Ticii cji j{.‘.cTii:ni8T.iG8 ugiihg a block dlvGIg 

.Tr;r:::iv;i pilt hi ATTPraiAT ciLJG.cTf: block 

29A 867 “^7v 23 39 666 



APiMIOACil 



Re :uoi L 18 (W. Pinch :P) (7) 

VULV.iGP CUiiU.:!JT ivKl^/nOLUHIP;; lU I:.;T,:0RKB op riONLUli-jJ: elements CONTiECTl'iD in pain^llel 
yOI.T CinUrlH-T K.JTIOU}; NON’LIIHLIR KLrlV.l'iNT COIINE P.‘G<i>.Ll^CL 

376 



954 



361 



412 



192 



674 



206 



Request 19 ' */• Plrichnm) (7) 

r;s:o;;.; op ni-m^oxiiJiTiNG pr-: 1 ; p.cy niiHE reutionbhir-^ por restbtj^/e ind;jctx'/e .jib uegigtivT’: c a ivvciii 

APPROXIM PlIE/UEtJC P1L\S lil^SISTIV INDUCTI (REBIATiy) CAPACIT 

15 "90S 251 393 251 341 



C I RCNITO 
ClilCuMT 

345 



Ue ri-jst 20 (l.C. Bain) (9) 

op ll-iPlO PU:CTl;./riO:;B in the rjaXTlIS magnetic PIPLD and TiIi-;iR relation TO THN HiOPAGATION OP 



0!i.:ER7\T 

200 



PUICTUAT 

105 



H/JITH 

693 



magni:tic field 
166 703 



H{OPAGAT 

231 



fiYDROlrlAGNPTIC .^.710 IN T!;:-: pyO.’I 
PVOROM.G .7A7 -; PaOGP:G*;R 

125 954 95 



Request 21 (./•C. Bain) (6) 

JiiiiG; .L V:J^Ii.TIOnO OP PUJCTh./riGiiS IN THi^ E;KRTii;] AAGNETTC PI ALL 
Oi;::c;.L v..inv/;i0 FlNCTiAAT P.'JiTH i^agnetic field 

372 32;. 105 693 166 703 



Request 22 (E. DunTord) (6) 

OlHE.aC 1. iii...;.:oN]G .i:;.aY;UO op T! 1E E.EmiC magnetic field 
.H AERLCi. rL^u'ONIC .Ju.LY ilJiTi! i'AGrA.TlC PI ;Lj) 

2/7 199 650 693 TUe 703 



Reqiiont 23 (E. Dunford) 

de:!Iv..t: ji: op cou^oiluto of th 

DEi{.l7P;’ GCMiA^NEi; 

69 50 



( 6 ) 



ELi'iCTRICAL CON'DlCTIViTY 
ELECTRIC CON DLiCT 

34 



ji: t.. 



350 



UPPER ATNOBi^irEiOE 
UPPEil ATIriOOFHE 

813 21 



Request 24 
■ME EPPECTE Oi 



Rishueth) (7) 

EOLA FLRlEo ON TEE ABEOEiTIu? OP COOHIC iL.DIO NOIEE 
GOL.R F E'Ea ^.BEOR COCNIC H.IDJO KOIE 

794 705 693 515 771 900 



IN TRE ICKOCnGHE 
lONOEi-HE 

136 



Request 23 (E. Dunford) (6) 

DETERMn.ATION OP TEE CHEiEOT fiYETEEB IN THE UPPER 
O' NJENT BYCTEM UI'RA 

361 632 81 3 



uTNOGPirErE DURING ;iAGI:ETIC OTORIrlG 
ATNOon 0 -: M\GN i:ti G BTORM 

21 1 66 803 



;ic fUtist 26 (D.L, Croom) (o) 

OBBERP/.TiCNa OP NUN DURITIG ECLIlABi:;; GIVIriG TiH: DIETiaBUTiuN OP BOURC/IS OU T;iE DT3C IV THE HlCRO.AvV'E PEJ^’GE 

OBEERV./i :rUN uGLIPG DIETRIBU SOliP.Ci: DI3 HICROE.'.V iLEiG 

200 993 330 80 613 970 177 917 



Request 27 (D.L, Croom) (4) 

OB-iERYATTOMO OF Till SUN USING iUJ)IO UnTEHFEUOUETKRG 
CBA.ERV/Pr SUN IGADIO IMTERFER 

200 993 771 145 



Request 28 (D.L, Croom) (7) 

UATIONS GU/EiiNING Tli]-; PROPAGAT I. N OP EKICT.'iOlGlGirETIC ;U:d irYDROI,AGNKA’IC .AiVE 
FROPaGAT ElA'CTROIu if/DROhAG 

231 86 125 



EEliAT 

696 



Vr'AVE 

954 



IN Tin-: GOLJl CORONA 
GOL’iR CORONA 

794 



514 



Request 29 (D.L, Croom) (‘ 

EfrillEATEG OP TIDi' DEI-^GITY OP lOJiVK-ATlOL 

D/h:git ioniz/;ti 

d 19 ' 153 



..Ad A'ENPKIL^TURE IN TIC; SOLAR COROia 
TEIrlF’lUAT S0L4R CORONiA 

29S 794 514 



(4) 



Request 30 (D.L, Croom) 

PLiGAGE G:-:i;D ABSTRiACTS ON TrG SOURCE SPECTRiA OP LIGHTNING DISCIPwRGES 

SOURCE GPECTR LIGIITNIN DISCH/iRG 

616 616 163 76 



110 



Request 31 ( Cano oiled) 

PI.Ki\GK SEND -iSSTRACTS COrmECTED 
CONNE 
67^ 



(5) 

V/ITH THE OCCURRK14CE OP A SECONDARY 
SECONDAR 
262 



OZONE LAYER AT NICHT TIME 
LAYER NIGHT TIME 

728 751 9A2 



Request 32 (D*M* Yates) (A^) 

Tine EFFECT OF SI/ALL DISTORTIONS IN Tlfli) SURFACE OF A CAVITY RESONATOR 
DISTORT SURFACE CAVIT RESONATO 

370 2i69 668 255 

Request 33 (^7.C, Bain) (3) 

THE DPJTERMHATION OF THE ORBITS OF INDIVIDUAL LEETEORS BY RADIO METHODS 

ORBIT METEOR R/iDIO 

755 771 



Request 34 (V/,C. Bain) (5) 

THE DETERMINiVnON OF ION M;\Gf;KS IN THE lONOSPHliRI-; BY THE STUDY OF BACK SCATTERED RADIO WAVES 
ION lONOSPHE SCATTER RADIO WAVE 

979 156 452 771 954 



Request 35 (V/,C. Bain) (9) 

THEORETICAL STUDL^JS OF THE SOUiiCE OF HIGH FRECjUETlCY RADIO V/AVES EMITTED FROM THE PLANET JUPITER 
THI'JOR SOURCE HIGH FREQUENC RADIO V/AVE EMIT PLANET JUPITER 

B 07 615 879 108 771 954 86o 586 400 

Request 36 (V/,C. Bain) (5) 

SII.TJLCANEOUS OBSl-JftVATIONS OF V^IISTLERS ;j;D LIGHTNING DISCHARGES 
SBIULTAN OBSERVAT . V/HISTL LIGHTNIN DISCHARG 

269 200 651 76 



Request 


37 (W.C, Bain) 


( 10 ) 














VARIATIONS 


IN THE HEIGHT OF REFLECTION OF 


L0\! 


OR VERY LOW FREQUENCY 


RADIO 


WAVES 


IK THE PERIOD 


BEFORE GROUND 


V/\RIATI 


HEIGHT REFLECT 




LOW 


(loyOfrequenc 


RADIO 


WAVE 


PERIOD 


GROUND 


324 


544 444 




982 


( 982 ) 108 


771 


954 


582 


541 


SUNRISE 


















SUimiSK 


















466 


















Request 


38 (ll» Rishbeth) 


( 7 ) 















THE USE OP IONOSPHERIC CROSS MODULATION IN THE DETERMINATION OF lONOSPtlERiC ELECTRON DENSITIES AND COLLISION 
CROSS MODUUT lONOSPHE ELECTRON DENSIT COLLISIO 

677 407 156 87 519 44 

FREQLTENCIKS 

l*T?Ev'iUENC 

108 



Request 39 (H, Rishbeth) ( 4 ) 

J.iEASuREra;TS OF lOHOSITORIC DRIFIS N1*DVR THE EQUATOR 
:.'iFvSUR lONOS™ DRIFT EQUATOR 

565 156 692 379 

Request 40 (H. Rishbeth) ( 5 ) 

THE USE OF AILVLOGUE COMPUTEIiS IN UPPl® ATMOSPHERE THEORY 
/iN/lLOGUE COIJPUT UPPER ATMOSPHE THEOR 

10 512 813 21 807 

Request 41 (H. Rishbeth) (5) 

THE USE 01' LUN/iH R/iDIO REFLECTIONS ITJ INVESTIGATIONS OF THE NATURE OF THE MOONS SURFACE 
ttnvt iUa)io REFLECT MOON SURFACE 



mi 
983 771 



444 



899 A69 



Request (D.A., Bryant) (9) 

WH/iT REFERENCES ON COMPARISON BETWEEN GROUND LEVEL AND HIGH ALTITUDE BALLOON COSMIC RAY RESULTS 



REFEHENC 

241 



COIAPARIS 



GROUND LEVEL 
541 730 



HIGH ALTI 

879 826 



BALLOON COSMIC RAY 
335 515 W? 



Request 43 (d*A, Bryant) ( 3 ) 

VJi/iT REFKIUiKCKS ON MODEL EXPERIMENTS ON AURORA 

REFERENC MODEL AURORA 

241 747 “ 49 ^ 



Request 44 (l>.A, Bryant) (6) 

\’fih\T Rl'JFERENCES ON HriEGRAL SPECTRUM OF PRIIAARY COSMIC RAYS 
REFERENC D'lTEGRAL SPECTR PRIIiARY COSMIC RAY 

241 141 616 432 515 W 

Request 45 (Cancelled) ( 5 ) 

REFERENCES ON SOL\R AND GEOilAGNETIC EFFECTS OF COSMIC RAYS 
REFERENC SOLAR GEOMAOIE COSMIC RAY 






794 



114 



515 987 



ERIC 



111 



r^oquost 4^ (Cftncellod) (4) 

CODJM(i DI:;CG l‘’0H A-JALOGUK DTCrl'll/iL CONVBilTiiKr. 

COD .u;alooui-; dtoxt cot;7:-;!iT 

96ft 10 6S7 357 

Hanuoat 47 (J. MoDaniol) (3) 

I'’.-:i^ROi.L\GHETIC T’iOmJ . jU/CIJ FOR C0MR’'r:')R 3T0RKS 

KKRHQMAG C OMRI T 3T0R 

103 512 938 

Request 2+8 (J , i.jcDaniol) (u) 

OF DJFFKRSMTI.AL K^IJATIOKF, iiY COF.l-UTHlR 

liOU/l’iOD DIFFDRFH S.'-UAT CO:iHJT 

F71 71 696 512 

Request 49 (J# IvlcDoniol) (6) 

FFFicr :::cY oi* jigvtal co:.;H:Ti‘:i4i vmrsiu; co;.;H:'rvu3 in tihi: golctior of roundary value f^oblems 

,:ff;cek digit /xIialogfje comput solution bound/ji 

83 687 10 512 271 339 



Request 50 (J. McDo.n.iol) (3) 

.•iTKODS 0? EiUiOH CliECKlIIC IN DlGl^.J. COI.iHITKRG 
ERROR DIGIT COMHiT 

697 687 51 2 



Request 51 (4» Longden) (7) 

;g:vmcd;; of PRODucriiG minii-u.l I'PEtc ovra\ 

PROD r.rill.. NKT 

913 72^5 985 



LOGICAL FUNCTION 112 C/^NOrjICAL FORM 
LOGIC FUNCTION CANONIC FORM 

735 109 340 872 



Re<|UOSt 52 Lonr,dGu) 

VK!TIitG-:TTC UNITS AS PG'/iUIiUlD IN 
UNIT 
947 



( 6 ) 

DIGITAL COMl^'TEK INCLUDING SHIFT 
DIGIT cc:.:iuT shift 

687 512 788 



REGISTNJNi SKP.iAL /JID P/Jt/vLUEL ADDER 
P.4;-/»r*I^L t\DU 
206 962 



Request 53 (C.H. Davis) ( 6 ) 

j:.;‘0;lt4.TISN ON Vtii:: D.-SIGN OF LOV/ DRIFT Tit‘d23ISTOU iJ.iPLIFIEHS 
INFORi.u.T D.-.SIGN LCV; DPJl'T TRiii:3I3T ASHJF 

136 520 982 692 514 491 



Kequent 54 (C.II, Davis) (5) 

u:c ifXGif CURitSNT TiFu:SISTOR SVaTCHKS 
INr’ORWiT HIGFi CUSIjENT TiDuNSIST 3V/ITCH 

1 36 c79 361 31 4 630 



Request 55 (C,H« Davis) ( 6 ) 

:uiA::j;./noN ok d..sign of tbeuivjsh;n :.ojlti flexing circuits 

XSFOf:.-/:' design TIF.EDIVI UJLTIPL CIRCUIT 

136 520 942 82,9 i.C9 345 

‘Request 56 (C.H, Davis) ( 3 ) 

JJTAIi;; OF AV-JLIBk*: LOV; VCL-LtCE CAPACITOP.S 
LO,/ VOLT C.JVCIT 

982 '^51 



Ro'^uc.''.t 57 


(C.H. 


Davis) 


(9) 






DI. -.i.L, OF DIRECT 


CGUF4-: 


D FLIP FLOF>fJ 


TO RTv^CTlON 


VITH THE M/'0(IMRri VIRlylTK^NS 


IN TiX: VILUES OF TIiS CIRCUIT 


D . j Gi 1 D i . i-". CT 


COUPl. 


FLIP 


ACTION 


I<4lXIH VAItLVnO 


CIRCUIT 


■j20 525 


676 


865 


109 


740 324 


345 


COKi'ANENTS 












co:.,rui:Er: 












50 












Request 58 


(a.a. 


Hill) 


(V) 







; l-IASE SlJPinY IIlFOtN-uinON ON TIH*; PSaFCHMJICH OP TYPICAL H/lGNETXC FllJvi MEMORY SYSTEMS NITH CIRCUIT DL\GI4J-:S 

Ii;F0:8.LKT PHRPORMuI til^GNKTIC FlliJ MEMOR SYSTEM crSCUIT 

136 210 166 742 632 345 

Request 59 (^-.A, Hill) ( 6 ) 

I ..0;,LD LIK.: DETAILS OF THE /OPK \a!ICH ILAS BKrIJ DONE TO EXTEND THE FREOUEIvlCY lUJv'GE OF MuiGNETIC /J^FPLIPTERS 

V/ORK EXTETFD FliROUENC R/dJG M\GIFETIC AMPLIF 

6 534 108 917 166 “T 91 



Request 60 (A. A. Hill) ( 6 ) 

T ODLD LIKE TUFOid.^tT] OK OU THE R/JICrK OF STATIC REL.YG 
ir'IFORlAiT R/’iNG STATIC RELAY 

136 917 621 774 



SUIT/vBLE FOR USE AT HIGH SVaTCHING Ri.THS 
HIGH SV/ITCH 
879 630 



Request 61 (A.A, Hill) ( 5 ) 

I .... JNTEREJVi’ED IK CKiCITTHY CAFABL:: OF GENEltVIlNG EXTREtELY NilRROV; PULSES 
OXliCUIT GENERAT EXTREH NARROW PULSE 

345 388 535 572 " 7 ^ 



112 



Request 62 (A,A« Hill) (/|) 

PLSASi: GUPPLY IIlFORI.I;iTIuu OK TIIK Tli'-iCRY ;a^ID USE OP PARiUlEmiC Al.iPLlPIERS 
niFomivr Tin-:oR r\ra?,{et amplif 

136 807 419 491 



Request 63 (P.A, Bricco) (^) 

T:E OYKT-IEOI;-. of iroTV/ORKr> V/ITII 0A!,IPLKD DATA TliivNSFER FUNCTIONS 

SYNTHESI flS'TWORK OAMPL DATA TIUUISFSR FUliCTION 

294 412 779 848 311 109 

Request 64 (P#A, Brig/;s) (6) 

Tli-i TliECRY OF STABILITY IN RELATinN TO COirTUOL OYSTEI/.S WITH RELAY ELEMENTS 
TliEOR 3TAB 7.LI CONTROL SYSTEM INULAY EliKMENT 

807 “^1 336 632 774 376 

Request 65 (P^# Bri^’lfis) (7) 

THE USE OF DIUTT/iL COMFJTERS TO OBTAIN POVR’^R SFECTRAL ANALYSIS OF NUMERICAL IViTA 
DIOIT COMl^'T POV/ER SPECTR ;a,'ALY NUMJORICA H\TA 

“3TT" 763 61 6 658 1 99 8/t8 

Request 66 (Cancelled) (8) 

CIRCUIT EL/;:i:::TS with zero 1.U':M0UY NCK LDILFiR ClI/vR/iCTERISTICS /JJD VARIi*vBLE PAR.UIETKRS 



CIRa’IT ELEIvniNT 


ZERO I.IEMOR 


LlNliiR Cil/UlilCTE 


V/tRIABLE 


P;JUVI.CET 


345 376 


961 7i^ 


559 39 


323 


419 


Request 6? 


(P*A. Bri/;f;s) 


(Q) 







mL'TED CIRCUIT DESIGi; FOR A R'aNDC^M BILSE GilJERATOR OF LOW ITO )UENCY 
raiNT CIRCUIT DESIGN Ib'JIDOM R^LSE GEIU-IRAT LOW FT(E.‘UENC 

■ 108 320 595 768 388 982 108 



Request 68 (D,A. Bryant) (5) 

xREf::renc.:s on elicctric field theories of the aurora 

RSFi-IREI^IC KLiilCTUIC F1;-:LD TllEOR AURORA 

241 34 703 807 496' 



Request 69 (C, II, Davis) (3) 

P;^T TR.\N3IST0R COUNTERS 
FAST TRi'»l^SIST COUNT 

864 314 675 

Request 70 (E.A* Newman) (4) 

liOW PASS liiTTICE FILTrH^S 
IDW P/i3 IWiTTICE FILT 

982 905 ^+02 867 



Request 71 (D.M. Yates) (7) 

SBlILWilTIES BKr.AiSn THE DIFi*U-\CTiON THEORY OF .CLECTRO?r>.GNETIC WAVES /U'lD THAT OF ELECTRON STREMI3 
SIMIUJI D IFFRiVC T THROR ELSCTROM V/AVE ELECTRON STREAM 

457 72 807 86 954 5? 624 

Request 72 (D.M, Yates) (5) 

THE USE OF COMPLilX V/a<Lu-.LES IN THE THEORY OF COMMUNICATION NET'.IO:IKS 
COMPLEX V/iRIilBLS TIIKOR COMMJNIC NETWORK 

349 323 807 45 412 



Request 73 


(D.U. 


Yates) 


(5) 








til: beiUviour o 


F BK tM 


OF C^UiHGE 


D PiLRTICLES 


IN THE niESENCE OF PIANE 


CONDUCTORS 




BFWU.i 


CiixvRG 


P'lRTICLE 




PLANE 


CONDUCT 




823 


669 


208 




759 


350 


Request 74 


(D.M. 


Yates) 


(7) 








PREDICTING TIE 


PATIiS OF 


EUiCI’ROI'IS 


MOVING IN A 


v;iRYING U/.GNETIC 


FIELD 




PREDICT 


PATH 


EIi:CTRQN 


MOVING 


VARY I.IAGMETIC 


FIEIJ) 




431 


906 


87 


571 


949 166 


703 




Request 75 


(J.R. 


Parks) 


(6) 








ACTIVi: COHST/JJT 


VOLT/lGE 


TPuJISFORMER FOR SIGN/iL DISTRIBUTION 






ACTIVE COKST/J^ 


VOLT 


TRx‘ii:SFOR 


SIGNAL DISTRIBU 






W 352 


959 


312 


613 


80 






Request 76 


(J.R. 


Parks) 


(4) 









ACTIVE AUDIO FirEf)USI'ICY FILTER \.1TH VARI/vBLE CUT OFF SLOPE 
ACTIVE FIU’IQUFUC FILT V/kRIABLE 

IS 7 1 08 867 323 

Request 77 (J.R# Parks) (4) 

VVRIABLE ULTiA HIGH IREOUENCY ATTENUATORS 
V/vRIABLE HIGH FREOUENC ATITINUAT 

323 “5t9 108 23 

Requeat 78 (J.R, Parks) (5) 

rtilll/nf OPERATION OF FESDIiilCK TIME BASES 

_^Y_.[IiT OPERATIO liTEDBACK TIME B/iSE 

201 101 9A2 852 
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Roqucr.t 79 (P#J# Pobqoo) (3) 

vyoiT/J3Jj% cAPAci’i.va;cK 
VARL\BIiJ CAPACIT ilWhlV 
323 3A1 

Ro quest 80 (P#J# Pobgoo) (3) 

TllUIBiriTOR BV/IWP GWPjllbiTORr, 

TWM'iVi 1ST SVfi-PiP G iiiiKUAT 

31 A 806 3^8 

Request 81 (P.J. Pobgoc) (3) 

adv.u;t..g:::8 op p;jLw.::n’n.Tc .uwP],ipL';rg 

.vi)V;jJTAG PgiaiiCT ..InPLlP 

5 M9 A91 

Request 82 (F»J, Pobgeo) (3) 

0PP1..;]SING LllfK/iR XI3T'./0RKS 
OFriM LUiK/tR R::T./0iiK 

75 A 559 M2 

Request 83 (P.J. 5'obgec) (a) 

TU/'JiSISTOR iUi;;::) SPblTTIAC GIRCUTTS 
TliiR.SlST m^y GPLITT CIUCiriT 
31 A 908 61 8 3A5 

Rcfiuest 8/j. Newman) (5) 

PU’l\?>7, SNPV-bY L.POlUtVTION R-j^TT;..:' T TC Tim USr: OF SUllPACR FiU';TREATMENT to PRKVrJ^T SECONDARY EI.;!.:iS3I0N EFFECT 
UiPOiaAT iiURl'ViCE SECOKDiVH EI/JSSI 

136 y+^9 262 531 

IN V/J.Vr:S 
V..LVE 
81 i,. 

Request 85 (E»A» il«r*vman) (7) 

I ..'1311 TO :iAT/; D.-iTA ..ROVT TIL: DirSICN OP P.ECiVai IC.TL MID PASS FILT :US FOR GOOD PASS Clb’ClA CAUSTICS 
DATA D ::;IGN I.L'CILANIC B/iND filt pass cii/ji/.cte 

bis 520 173 831 867 905 39 



Request 86 (2. A, Newman) (6) 

\/iiE?iiBY TA^.s:.;i3;;i>,.N ;.t nigh ppj:; :u:-:r;ci.:s is /.i’F 2 CT;':D by v.'e.ather /gid time of day 

:.;ACHAi;i3 TRAI.SI.I lilOi; FNN 'UENC TIME DAY 

17A 476 879 108 942 969 

Re (pest 87 Newman) (2) 

coNLD YOU PLV..;:: civr, :.3 ._,vicr;s .-.bout tiu-: possibidities of getting rectification using mktalijg de^/ices 

RECTIF METAL 

599 743 



Request 88 liewmaji) ( 7 ) 

I ..ISii TO C.J.CULTTE TiiE DIDUCT/J^C.; 3ID LOSS Hi COILS tADN USING PRIN'TED CIRCUIT OR OTiClR MINLVrURIZi\T10N 

G-.LCUJAT HlDUCT.-’Ui LOSS COIL PRIIiT CIRCUIT MINLvTuR 

35 134 393 841 764 345 161 

TD:vi.S* A SUxTiUiJjE ^{TICTxj PLE/1.3E 
Request 89 (R.S* w’atson) (5) 

;j:alysis op i:ohlz;-UvR SYsrais usDiG PHiisi-: plwe tiochiii »ues 
;u:aly nqi;lh:ea system fnas plane 

658 192 632 908 739 

Request 90 (R.S* V/atson) ( 7 ) 

MINI/vTUTiE LOV/ NOISE HIGH GAIN HIGH IMi^DiGiCr: ;J>U'LIFIER3 
MHil/VTUR LOV/ NGI’IE GAHvi HIGH IMP1<:D.\NC /U.IPLIF 

181 982 900 874 879 128 A 91 



Request 91 (R*^» V/atson) ( 5 ) 

CONTROL CliTUtiCTERISTICS OF S.U1PLING SERVO SYS'TEMS 
CONTROL CHiVR-TCTE SVuMPL SERVO SYSTEM 

356 39 779 782 632 



Request 92 (R.S, V/atson) (5) 

POV/ER SHCCTILAL DI21SITY DISTRIBUTION OBTAINED USING iJ'iALOGUE TECHTUOUES 
POAER SPKCTR DEI'iSlT DISTRIDU /GiALOGUE 

763 616 519 80 10 



Request 93 
HIGH STABILITY 
STABILI 

y^i 



(R.o, V/atson) 

HIGH INPUT IMPED^^GICE 
HIGH HOTT I1.:FED/J^C 

879 72 A 128 



(7) 

T1U/»N3IST0RISKD /GIALOGUE IJ.IPLIFIER 
TlbUiSlST /GiALOGUE MfPLIP 

314 10 491 
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Roquost 94 (P#U« Stuart) ( 6 ) 

;::z:c:tug:.ic if.vvr o? oUp:;rcondijctor SHovmiG l dlscoiitii:utty kt swPr3^coriDUCTin& critical 

i^ioECTUoii Gi;r»i^,RcoN Di 3 co:iTi (:iu!-:::ucow) critical 

n? o7G 291 77 (291) 64 

T;-;:.:piiRiVTUiffi 

T-XimVT 

296 

Rcq\iest 95 (P«U« Stuart) (B) 

/., u'BSTIL.CT TJ[r: P.GLD DiSTia Bl-TICi; SiriliiObT<J;H;G A CIMGKJ) TI!IN CIRCUL>uR J)ISC JLCTH.’G ON /J^ INFINITE 
PI.aD DTfJVRIUi; CHi\RG TIIL’J CIRCUL/\R DIB UIFDIIT 

703 30 669 941 AO 970 395 

DI-CiVJC'rHiC SUB 

DINLECTU 

70 



Roquoot 96 (P«R« Stuart) 

TUNNEL DIODE CONSTUIIC'OION AND ITS 
mTOL DIODE 

6 li 2 689 



(4) 

ELr:crxiICAL CIL/JDVCTERISTICS EXPUINED 
KUiCTRlC CIU-JUCTK 
04 39 



Ronucst 97 (P#R* Stuart) (5) 

Eu:cTRCi;ic D.'SisrrY of stat-is ..t til: sbuface of a semiconductor compand v.Trii tii\t at depth 

ELECTRON DICTABTT SIH^FAC K SEMICOND DSITH 

87 519 ~lS9' ~2^ 685 

Rcnuost 93 (3-*U, Stuart) (6) 

1L*::*,IST..'/ITY OF I-CTALLIC Tli'lN FILMS IL^UTED TO SURl-’ACE ROUGILIEGO 
R.A')LST1V Mt:TAL THIN FIU! SJRFACi: RQfr jGHHEs 

251 743 9^.1 866 469 259 



Request 99 (D.U. Yates) (9) 

THE Pin2i0i.iEN0N OF a'-J)Ii\TI0N CMUHED BY CiLARGED PiVRTlCLES bOTOG IN VARYH^G ELECTRIC AND MAGtrETIC FIELDS 

Pn2i:OL2N a^L DKTIO CHARG PARTICLE MOOTING VARY EIECTRIC IvV^Gi^STIC FKLD 

213 235 ^69 208 571 949 84 166 703 



o 
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APPENDIX B2 



Numbers and distribution of key-word in reques1>a G-99 
( ) = cancelled request 

Request Numbei 01 23A-5 6 789 



0 


6 


(10) 


5 


6 


9 


5 


(3) 


5 


( 4 ) 


4 


10 


7 


7 


7 


5 


3 


7 


8 


6 


7 


7 


20 


9 


6 


6 


6 


7 


6 


8 


4 


7 


5 


30 


4 


( 5 ) 


4 


3 


5 


9 


5 


10 


7 


4 


40 


5 


5 


9 


3 


6 


(5) 


(4) 


3 


4 


6 


50 


3 


7 


6 


6 


5 


6 


3 


9 


7 


6 


60 


6 


5 


4 


6 


6 


7 


(8) 


8 


5 


3 


70 


4 


7 


5 


5 


7 


6 


4 


4 


5 


3 


80 


3 


3 


3 


4 


5 


7 


6 


2 


7 


5 


90 


7 


5 


5 


7 


6 


8 




5 


6 


9 



Distribution 



Number of Keywords 



1 2 3 



k 



678 



9 10 




93 requests. 

Mean 5o75 key-words 
Mode 5o5 key-words 

Standard Deviation 
1 o 6 key-words 
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APPENDIX B3 



Subsets of Requests 



(i) Cancelled Requests 

7 of the original 100 requests were cancelled at various stages of the 

worko 

Request 1o 'Super-regenerative* is a single word in the dictionary and the 
abstracts, but hyphens were coded as spaces throughout our work. 

Request 6o ’’Testing" is not in the dictionaryo 
Request 8o Request garbled in typing® 

Request 31 o 'Ozone* is not in the dictionary® 

Request 45 o Requestor not available for assessments® 

Request 46® Dictionary mistakenly does not accept plural of 'disc'® 

Request 66® *Non-linear*, as with Request 1 . 

(ii) The '55^ set (report VI®2o6) 

The following 55 requests had one or more words underlined 
0, 2, 4, 7, 10-12, 14-19, 22-24, 26-30, 32, 33, 35, 36, 42-44, 47-50, 58, 59, 
61 , 62, 64, 65, 67, 68, 71, 74-79, 8l , 89, 94-99 p 

(iii) The '34' set (report, VIllo2) 

The following 34 requests retrieved only 0-4 relevant documents in Run 13 
at K' ^ 50:- 



3, 5, 7. 11-14, 35, 38, 40, 43, 49-51, 54-56, 6o, 64, 65, 67, 71-74, 77, 78, 
81, 87, 88, 92, 94, 96, 99p 

(iv) T he '17* set (report, VIII 0 I ) 

The following 17 requests were reckoned less well formulated: - 

5, 12, 13, 34, 43, 52-55, 60, 62, 76, 84-86, 94,96. 

(v) Four Request Sets of Increasing Cenerality (report VHlo4) 

G-en 1 (O -8 kno7m relevant documents) (26 requests) 

5, 7, 11, 12, 14, 32, 34, 37, 38, 40 , 43, 49, 50, 51, 54, 55, 56 
65, 67, 72, 73/ 77, 83, 88, 92, 94 
Gen 2 (9-17 known relevant documents) (2l requests) 

3, 9, 13, 21, 22, 33, 35, 36, 42, 57, 59, 60, 61, 63, 64, 76, 87, 

91 , 95, 96, 99 

G-en 3 (l8-29 known relevant documents) (23 requests) 

2, 18, 19, 20, 23, 26, 30, 41, 44, 58, 69, 70, 71, 74, 78, 80, 84, 
85, 86, 89, 90, 93, 97 

Gen 4 ( 3 O -84 known relevant documents) (23 requests) 

0, 4, 10, 15, 16, 17, 24, 25, 27, 28, 29, 39, 47, 48, 52, 53, 62, 68 

75, 79, 81, 82, 980 



O 
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(vi) The 'Sub.iect Index* set (report, VII,4 ) 

Subject Index searches were carried out for requests 

0, 18, 22, 38, 41, 57, 58, 70, 74, 75, 83, 94« 

(vii) The 'Virtually Exhaustive Search* set (report, ^11.4 ) 
This involved requests 

10, 38, 91 o 



i 
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APPENDIX B4 



Mnemonic 


Run 


Formula for Coordination Value 


Basic Key Word Stem Strate^r 


KWS 


13 


Number of Key -word Stems common to abstract and request 


KWS 


with Automatic Weighting 


AWKWS 


22 


Weight = 2 if word not present in 8(^ of output for 
Run 0 (ioeo Run 13 at K = ^ 0 , K* 35) 


Descriptor Strateries 






Number of Descriptors in common, using:- 


ARM 


16 


A R Meetham*s non-overlapping clusters from G-3j (C2) 


MCS01 • 


14 


P Vaswani’s 1178 overlapping clusters, (C3) > 


MCS11 


20 


P Vaswani*s 725 overlapping clusters, (C5), 


JRJR 


19 


R J Reason® s 702 slightly overlapping clusters from G-3j 


Descriptors 


with Representation 


ARMSR 


17 


2^0 Run l6 Representation + Run l6 Coordination 


SR14 ■ 


6 


2*^0 Run 14 Representation + Run 14 Coordination 


PDR14 


11 


2^0 Run 14 Proportional Representation ^ Run 14 






Coordination, where a request word is represented 
if a third of its descriptors are assigned to the 
abstract 


Association 


Strategies 


M&3 


15 


Number of Request Words associated with some abstract 
word via &3 


FAG4 


18 


As 1 5 but with G-4 


EARG4 


23 


Number of Associations via G4 


Combinations of Q?wo Strategies 


13T14 


9 


Run 13 Coordination times Run 14 Coordination 


15W14 


28 


2^0 Run 13 Coordination + Run 14 Coordination, 
effectively subdividing the strata of Run 13 


Strategies with Underlining 


U14 


21 


Run 14 with underlining 


U11 


24 


Run 1 1 with underlining 


U13 


25 


Run 1 3 with underlining 


Searches Largely Manual 


SI 


26 


Subject Indexes - No Coordination 


EXHSEA 


27 


Virtually Exhaustive Search - No Coordination 
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Discarded 



^ As Run 16 but ignoring isolated words as descriptors* 

Note 1 o Runs 023456781012 were 

Runs 13 15 l6 17 14 19 20 21 ' 22 respectively at K = 50, rather 
than K* - 50 o 

Note 2o Run 24 is fully asse- ed up to and including K = 1 8, K* “15 

25 K = 27, K’ “17 

28 K = 42, K' “ 40 

but careful use may be made of these runs beyond this point. For example. 
Run 28 and Run 13 yield identical total numbers, or nearly, of relevant 
documents at K' “ 50 as well as at K' = 20 , 30 , 4 O 0 
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APPENDIX B6 



Computer Implementation of Indexing and Retrieval Operations 
Matrix manipulation of data 

A matrix is a rectangular or square array of numbers or data elements. 

We have dealt with them throughout this report, but we have not mentioned 
operations upon matrices. One used extensively in the organization of 
computer programs for indexing and retrieval is that of multiplying two 
matrices. The prodi^ct, AoB, of two matrices is defined only when the number 
of columns of matrix A and the number of rows of matrix B are equal. The 

product is then another matrix having as many rows as A and as man^'’ columns as 

B. The element in row-i and column-j of the product is obtedned as the sum 

of products of corresponding elements in row-i of A and column- j of Bo This 

is illustrated in fig. B6 o1 . 

A.B 




3x2+0xi +7x2+1 x3+5xl 0+9x0=33 



Pigo B6 o 1 Illustration of matrix multiplication 

Consider this process in the case where A is a b nary abstract/word 
matrix (i.eo rows corresponding to abstracts, columns to words) and B is a 
binary word/request matrix. Row-i of A is then a string of O^s and 1 *s, 
the 1 *s representing the particular words in abstract-i, and column- j of 
B is a similar string of 0*s and 1 *s in which the 1 *s represent the 
particular words occurring in request-j . The value of the element in row-i 
and column-j of the product will then be the number of words in common, i.e. 
the key word coordination level, for the austract and request in question. 
The product matrix contains the key word coordination levels for all 
abstracts with respect to all requests. The rows of this matrix correspond 
to abstracts and the columns to requests. The organization of our programs 
in the computer is such that the matrix of coordination levels produced in 
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each ruxL is required to have rows corresponding to requestso Having 
produced the matrix for run 13 as above it is therefore necessary to 
transpose it so that its columns become rows and vice-versao Eefore trans- 
ferring the lists of retrieved abstracts to the harvest file (Section Vo 8) 
it is also necessary to apply the necessary coordination cut-off level for 
each request (Section VI.1), rejecting abstracts at lower levelSo 

Three matrix products are formed in the case of a simple descriptor 
run such as run 14* In the first the abstract/word matrix is multiplied by 
a word descriptor one to obtain an abstract/descriptor matrix o In accordance 
with the indexing strategy this 5 s converted to binary form by setting a 
threshold value of one. We now have a matrix indicating the descriptors by 
which each abstract is indexedo The descriptor/word matrix (the transpose 
of the word/descriptor one) is then multiplied by the word/request matrix, 
again thresholding at one, to give a descriptor/re quest matrixo Finally the 
product of the abstract/descriptor and descriptor/re quest matrices yields an 
abstract/request one which, as before, requires transposing and application 
of the cut-off processo 

Hun 14^ an associative run using word connection matrix G-3 directly, 
involves only two products o First the abstract/word matrix is multiplied by 
&3o This yields a new abstract/word matrix which is converted to binary by 
thresholding its values at one (for strategy see Section VIol)o This is then 
multiplied by the word/request matrix to obtain the abstract/re quest matx..Xo 

The main operations used are thus multiplication, transposition and 
threshol dingo 

Data formats 

Two formats were adopted for matriceso With the first, used for 
storage on magnetic tape or for holding individual matrix rows in the core 
store, each row is stored as a block of successive machine words, the first 
of which specifies the number of words to followo The remaining words in 
a block specify the non-zero elements in the matrix row, each word containing 
the element value and column number of one element o 

Binai*y matrices with not more than 1 ,000 rows or columns could be held 
entirely in the core store by adopting a more economical format o Coliamn 
numbers of all the non- zero elements of a matrix are held, three per word, 
in a block of words and are accompanied by a kej indicating the stai‘ting 
point within the block of each rowo 
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Matrix multiplication 



With one or two exceptions all the multiplication involves binary 
matriceso For simplicity and processing speed the programs took advantage 
of this fact. Referring to figo B6o1 we see that the non-zero elements of 
row-i of matrix A indicate those rows of matrix B that must be accessed to 
obtain element-ij of the producto Furthermore computation of every element 
of row-i of the product requires access to these same rows of matrix Bo It is 
therefore economical to compute all the elements in a row of the product 
together o For binary matrices it is necessary simply to add together 
corresponding elements of all the selected rows of matrix B in order to obtain 
a row of the product matrix o 

Two multiplication routines are used working on this principle o One is 
used when both matrices in a product are small and are held entirely in the 
core store* The other requires the post-multiplier (the RoHo matrix) to be 
in the core store and reads successive rows of the pre-multiplier from 
magnetic tape* Both write successive rows of the product, which is generally 
non-binary, to magnetic tape* 

Matrix thresholding 

One program reads a complete non-binary matrix from magnetic tape and 
stores it in compact binary form in the core store after setting a threshold 
value one (the commonest value )o 

Another program accepts any stated value in thresholding a single row of 
a non-binary matrix held in the core store and leaves the result in the corOo 
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Coordination value cut-off 

The abstract/tequest matrices have 11,571 rows and 100 columnso 
Transposing a matrix of this size on a computer is a difficult and a slow 
job, the time increasing with the number of non-zero matrix elements. For 
this reason the coordination cut-off level for ea^h request is computed and 
applied before transposing the matriXo 

Two programs are used here* We wish to obtain an average output as 
close as possible to 50 abstracts per request by retrieving as near as 
possible to some fixed number^ K, of abstracts for each (see Section Viol)* 
The first program forms 100 distributions, one for ea^h request, of the 
coordination values (ioe* the element values of the abstract /request matrix) 
and from them computes the value Ko The second program makes two passes of 
the tape containing the abstract/request matrixo During the first the 
distributions are re-formed the coordination cut-off level giving an 
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output closest to K. is recorded for each request® On the second tape pass 
the matrix is thresholded using these separate cut-off values for the different 
columns and the resulting matrix is written on to magnetic tape© 

Matrix transpose 

Difficulty only arises in the case of very large matrices such as the 
abstract/request and abstract/word ones© We will therefore only describe 
the method used for these. The number of passes of the tape containing the 
matrix to be transposed depends upon the number of non-zero elements. The 
first pass is used to find the number of elements in each columno Knowing 
this and the amount of core store made available for holding the transpose, 
the program decides how many rows of the transpose to form on the next and 
subsequent pa* s of the tape© The requisite number of passes of the input 
tape is then made, each time forjiing a subset of the rows of the transpose 
in the available core store and then writing t!^em on another magnetic tape. 

Retrieval print- out 

The texts of all 11,571 abstracts are read from magnetic tape and 
dumped on to a magnetic disc store giving reasonable random access© Having 
ied the contents of the requesty^abstraot matrix for the rm in hand on to 
the harvesv file, the latter is now accessed to obtain the list of abstract 
numbers retrieved for each request© The text of each request, held in the 
core store, is assembled with those of the retrieved abstracts as they are 
called down from the disc© The complete output for each request is sent In 
the required format to a magnetic tape. When the entire output for ^ rim is 
written on tape another program is run which reads it from cape md sends it 
to the line-printer. The programs are organized in this way so that the main 
program, the inn time of which is drawn out by the vast number of disc 
accesses, does not monopolize the line-printer, inhibiting its use by other 
programs that may be running in the machine in parallel. 

Computer time required for indexing and retrieval 

It is assumed that the abstract/word matrix the word/request matrix 
and any required descriptor/word matrix are available in machine readable 
form© 

When a program l eing mn in time sharing mode on KDF9 terminates, three 
times are printed out: 

1 o the time for which the program actually occupied the central 
processor unit. 
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2o the total elapsed time during which the program was running in 
parallel with others, and 

3o the notional elapsed time, which is an estimate of the time for 
which the program would run on a machine without time sharingo 
In calculating the following times ( 3 ) was usedo 

We consider run 13^ the basic key-word strategy, and rm.is 11 and 23, 
the best single runs with which to supplement ito Run 11 uses a cluster 
strategy with proportional representation and run 23 involves expanding 
requests and abstracts with G-4, but does not use clusters (see Section Viol). 

Below we give the total times to run all necessary programs to produce, 
from the basic matrices mentioned above, a re quest/abs tract matrix (for 93 
requests): 



Bun 


Number of 


total notional 




matrix operations 


elapsed time 


13 


3 


1 6 mins 


11 


10 


164 ” 


23 


5 


69 " 



Although the machine times required, having produced a re quest/ab struct 
matrix, to obtain the final printed output of retrieved material are 
substantial, they would not be very meaningful figures to quote o This is 
because the economics of this .id of the process depend upon factors such 
as whether or not the computer is time-shared, what sort of backing store the 
document texts are held on and whether retrieved texts are printed on-line 
to the computer or off-lineo These times are also independent of retrieval 
strategyo 

In our case the texts of the abstracts are held on magnetic taj;e and 
are only dumped on to a magnetic disc when the appropriate program is 
runningo It takes approximately 17 minutes to transfer them to the disc, but 
as our machine is time-shared that is quite reasonable o In an operational 
environment the texts might reside permanently on a disc fileo The line- 
printer on our KDF9 can only be used in an on-line modeo Again, having time- 
sharing, it is not too unreasonable to print extensive quantities of output 
in this way, although it does monopolize the printero In an operational 
system one would try to arrange that large amounts of printing are prvoduced 
in an off-line modeo 




APPENDIX B6 (Continued) 



Evaluation Files and Propirams 

The Harvest File is the central information bank and is arranged thus on 
magnetic tape : - 

There is one block of information on the tape per requesto 
The block *s machine words are each arranged thus, 

Flag/strategy Numb er/Relevance/Coordination/Abs tract, 
ordered by strategy and then abstraoto 

The blocks are overwritable for later insertion of relevances, or 
other corrections o To lengthe n the blocks with the results of a new strategy 
the file is copied from one of two tapes to the other in turno 

The flag marks abstracts retrieved by no previous strategy© 

Flagged abstracts only are sent to request or to avoid duplicate assessment 
and reduce unnecessary output© The flags are also of use in counting, for 
example, the numbers of distinct relevant abstracts, or all distinct abstracts. 

The harvest file can be reduced to study subsets of requests or smaller 
cut-off S o 

Programs to Run and Evaluate a ly-pical Strategy 
Each step after the first represents a single programo They rely on 
store and on a single pass of each with each input or output tape, and 
take 3--1 5 minutes on KDF9o 

1 © Form coordination for each abstract-request, for example by matrix 
multiplication or merging two tapes, in order of abstracts© 

2o Print out numbers of abstracts in each stratum© 

3. List K’ for K=1, 2, 3 o....© 

^© Cut off 1 ) at largest K or K’ likely to be needed and transpose, i©e© 
rewrite 1 ) in order of requests. 

5o Suppress permanently cancelled requests as necessaryo 

6o Add new strategy to Harvest file, mark abstracts not previously retrieved 
with flago 

7* Print out new abstracts and send to requestors for assessment© 

8. Insert assessments of flagged abstracts via paper tape© 

9o On Harvest Tape, copy assessments of all unflagged items where known 
from an earlier run© 

10o Replace existing coordinations by 1 00 - K, where K = minimum K-value 
needed to retrieve stratum in which abstract occurs© This preserves the strata 
while enabling any smaller cut-off to be applied by a simple program step© 
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using appropriate K 



11 o Form one new Harvest File for each desired K', 
froni step 3 o 

12m Reset Harvest File Flags if earlier runs have had items removed in 
step 1 1 o 

13o Form other Harvest Files with selected requests onlyo 

14o Several Print-Outs, each displaying s 

For Fixed Cut Off , Each Rm :, Each Request 

The numbers of relevant, irrelevant, the fo precision and recall, also 
totals and average numbers of relevarit per request; the same after 
suppressing abstracts also found in a stated run (eogo Run 13) or runso 

15* Several Print-Outs, each displaying! 

For K = 1 , 3 o . . o o and One Bun 9 One Request Set 

K* 

K, K*, , totals relevant, irrelevant, unassessed, output, Overall 

Precision and Recall (report VHo^, Average Precision and Recall with 
Standard Deviations o 

Programs to display behaviou’:^ of strate^ for lar^e cut-offs 

160 Collect abstracts known to be relevant from all available sources. 

17o Ii:sert coordinations for given strategy from l)o 

180 For each strat’om display numbers of relevant from 1 6 ) , and of the 

remainder, assumed irrelevant, using 2 ), precision and recallo 

19o As 1 5 ) as far as Overall Precision and Recall, for K = 1 to 500 (say)o 
Other Programs 

2 O 0 For a given Run display request words represented by each abstract. 

21 o Display words associated with request words via some descriptoro 
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APPENDIX B7 



Performance of Successive Strata in Run 13(KWS). Table RCCOO . 



Unassessed documents, that is, those not retrieved by standard 
strategies or by the two manual searches, have been set 'irrelevant 
(c.f. VII. 4). 



Coord- 
inat ion 


Cumul- Rele- 

ative vant 


Irrel- 

evant 


%Prec- 

ision 




Output 




REQUEST 0 


5 


1 4 


7 


7 


50.00 


4 


1 0 1 


30 


57 


34.48 


3 


564 


8 


455 


1.73 


2 


1 952 


1 


1387 


.07 


1 


559 1 


0 


3639 


.00 


0 


11571 


0 


5980 


.00 




TOTAL 


46 


1 1525 


.40 








REQUEST 1 


0 


1157 1 


0 


11571 


.00 




TOTAL 


0 


11571 


.00 








REQUEST 2 


4 


2 


1 


1 


50.00 


3 


26 


8 


1 8 


30.77 


2 


23 1 


9 


194 


4.43 


1 


2034 


1 


1802 


.06 


0 


11571 


0 


9537 


.00 




TOTAL 


1 9 


1 1552 


• 1 6 








REQUEST 3 


3 


1 4 


4 


1 0 


2B.57 


2 


242 


7 


22 1 


3.07 


1 


2668 


4 


2422 


. 1 6 


0 


11571 


0 


8903 


.00 




TOTAL 


15 


1 1556 


.13 








REQUEST 4 


7 


1 


1 


0 


100.00 


6 


9 


2 


6 


25.00 


5 


32 


10 


1 3 


43.48 


4 


105 


5 


68 


6.85 


3 


300 


1 0 


185 


5.13 


2 


873 


4 


569 


• 70 


1 


3092 


1 


22 1 8 


.05 


0 


11571 


0 


6479 


.00 




TOTAL 


33 


1 1538 


.29 








REQUEST 5 


3 


7 


2 


5 


28.57 


2 


125 


2 


1 1 6 


1.69 


1 


1568 


1 


1442 


.07 


0 


11571 


0 


1 0003 


.00 




TOTAL 


5 


1 1566 


.04 








REQUEST 6 


0 


11571 


0 


1157 1 


.00 




TOTAL 


0 


11571 


.00 



O 




REQUEST 7 



3 


12 


0 


1 2 


.00 


2 


375 


3 


360 


.83 


1 


3701 


1 


3525 


.03 


0 


11571 


0 


7870 


.00 




TOTAL 


4 


1 1567 


.03 








REQUEST 8 


0 


11571 


0 


11571 


.00 




TOTAL 


0 


11571 


.00 








REQUEST 9 


3 


4 


3 


1 


75.00 


2 


53 


3 


46 


6. 1 2 


1 


739 


4 


682 


.58 


0 


11571 


0 


1 0832 


.00 




TOTAL 


1 0 


11561 


• 09 








REQUEST 10 


5 


5 


2 


3 


40.00 


4 


29 


9 


15 


37.50 


3 


1 1 4 


44 


4 1 


5 1.76 


2 


399 


17 


268 


5.96 


1 


2 196 


3 


1794 


• 1 7 


n 


1157 1 


0 


9 3/5 


.00 




TOTAL 


75 


1 1496 


.65 








REQUEST 1 1 


'J 


1 


0 


1 


.00 


4 


9 


1 


7 


12.50 


3 


70 


0 


61 


.00 


2 


546 


0 


476 


• 00 


1 


3073 


0 


2527 


• 00 


0 


1157 1 


0 


8498 


• 00 




TOTAL 


1 


1 1570 


• 0 1 








REQUEST 12 


5 


3 


1 


2 


3 3.33 


4 


22 


0 


1 9 


• 00 


3 


79 


1 


56 


1 .75 


2 


34 1 


0 


262 


• CO 


1 


2416 


0 


2075 


• 00 


0 


11571 


0 


9 1 55 


• 00 




TOTAL 


2 


1 1569 


• 02 








REQUEST 13 


4 


2 


0 


2 


• 00 


3 


19 


4 


1 3 


23.53 


2 


360 


7 


334 


2.05 


1 


2922 


0 


2562 


• 00 


0 


11571 


0 


8649 


• 00 




TOTAL 


1 1 


11560 


. 10 



3 

2 

1 

0 

5 

4 

3 

2 

1 

0 

5 

4 

3 

2 

1 

0 

4 

3 

2 

1 

0 

5 

4 

3 

2 

1 

0 

4 

3 

2 

1 

0 

7 

6 

5 

4 

3 

2 

1 

0 







REQUEST 14 


1 


0 


i 


• 00 


51 


0 


50 


• 00 


1664 


4 


1829 


• 22 


1157 1 


0 


9687 


• 00 


TOTAL 


4 


1 1567 


.03 






REQUEST 15 


4 


2 


2 


50.00 


24 


1 1 


9 


55.00 


232 


26 


182 


12.50 


13 6 1 


1 


828 


• 12 


3823 


0 


2762 


oOO 


11571 


0 


7748 


• 00 


TOTAL 


40 


1153 1 


• 35 






request 16 


1 3 


7 


6 


53.85 


43 


1 9 


1 1 


63.33 


224 


23 


15 8 


12.71 


857 


8 


625 


1 .26 


3003 


2 


2144 


.09 


1157 1 


0 


8568 


• 00 


TOTAL 


59 


11512 


.51 






REQUEST 17 


3 


1 


2 


33.33 


52 


29 


20 


59.18 


324 


1 9 


253 


6.99 


2452 


6 


2122 


.28 


11571 


1 


9 118 


.01 


TOTAL 


56 


11515 


. 46 






request 18 


2 


0 


2 


• 00 


1 3 


7 


4 


63.64 


107 


1 0 


84 


10.64 


6 1 1 


9 


495 


1 . 79 


2637 


6 


2020 


• 30 


1157 1 


0 


8934 


• 00 


TOTAL 


32 


11539 


.28 






request 19 


1 9 


1 6 


3 


84.21 


1 67 


1 0 


138 


6.76 


964 


2 


795 


• 25 


4059 


0 


3095 


.00 


1157 1 


0 


75 1 2 


• 00 


TOTAL 


28 


11543 


• 24 






REQUEST 20 


1 


0 


1 


• 00 


9 


2 


6 


25.00 


27 


5 


1 3 


27.78 


1 18 


7 


84 


7.69 


395 


7 


270 


2.53 


1414 


2 


1017 


• 20 


4097 


0 


2683 


• 00 


1157 1 


0 


7474 


• 00 


TOTAL 


23 


1 1548 


.20 









REQUEST 21 


5 


5 


0 


5 


.00 


4 


43 


6 


32 


15.79 


3 


275 


5 


227 


2.16 


2 


1 163 


2 


886 


• 23 


1 


3296 


0 


2 I 33 


• 00 


0 


11571 


0 


8275 


• 00 




TOTAL 


1 3 


1 1 558 


• 1 1 








request 22 


5 


2 


2 


0 


100.00 


4 


18 


7 


9 


43.75 


3 


243 


9 


216 


4.00 


2 


1 127 


4 


880 


• 45 


1 


3655 


1 


2527 


• 04 


0 


1157 1 


1 


79 15 


• 0 1 




TOTAL 


24 


1 1547 


• 2 1 








request 23 


4 


3 


3 


0 


100.00 


3 


30 


8 


19 


29.63 


2 


366 


1 2 


324 


3.57 


1 


199 1 


0 


1625 


.00 


0 


11571 


0 


9580 


• 00 




TOTAL 


23 


11548 


• 20 








REQUEST 24 


6 


2 


2 


0 


100.00 


5 


15 


8 


5 


6 1.54 


4 


70 


2 1 


34 


38.18 


3 


237 


1 6 


151 


9.58 


2 


792 


3 


552 


• 54 


1 


2759 


0 


1967 


• 00 


0 


11571 


0 


88 1 2 


• 00 




TOTAL 


50 


115 2 1 


• 43 








REQUEST 25 


5 


3 


2 


1 


66.67 


4 


1 8 


1 0 


5 


66.67 


3 


9 1 


3 1 


42 


42.47 


2 


66 1 


1 9 


55 1 


3.33 


1 


3354 


3 


2690 


• 1 1 


0 


1157 1 


0 


82 1 7 


.00 




TOTAL 


65 


1 1 506 


o 

in 

• 








REQUEST 26 


7 


f 

1 


1 


0 


100.00 


6 


2 


0 


1 


• 00 


5 


6 


2 


2 


50.00 


4 


24 


4 


1 4 


22.22 


3 


85 


5 


56 


8.20 


2 


440 


7 


348 


1 .97 


1 


2711 


1 


2270 


• 0^ 


0 


11571 


0 


8860 


• OC 




TOTAL 


20 


11551 


• n 








REQUEST 2/ 


4 


3 


3 


0 


1 00 • 0( 


3 


55 


1 5 


37 


CO 

. 

CO 

C\J 


2 


374 


14 


305 


4.3S 


1 


1621 


7 . 


1 240 


.5< 


0 


11571 


0 


9950 


»CC 




TOTAL 


39 


11532 


• 3< 



130 



2 

1 

0 

4 

3 

2 

I 

0 

3 

2 

1 

0 

0 

3 

2 

1 

0 

3 

2 

1 

0 

5 

4 

3 

2 

1 

0 

6 

5 

4 

3 

2 

1 

0 

4 

3 

2 

1 

n 







REQUEST 26 




17 


4 


1 3 


23.53 


6 


1 22 


12 


93 


11.43 


5 


599 


55 


422 


11.53 


4 


24 9 1 


2 


1890 


. 1 1 


3 


11571 


0 


9080 


.00 


2 


TOTAL 


73 


11498 


• 6 3 


1 










0 






REQUEST 29 




2 


2 


0 


1 00#00 




34 


a 


24 


25#00 




29 1 


43 


2 1 4 


16.73 


6 


1667 


0 


1 576 


« 00 


5 


11571 


0 


9704 


.00 


4 


TOTAL 


53 


115 16 


.46 


3 










2 






REQUEST 30 


1 


7 


5 


2 


7 1.43 


0 


70 


1 2 


5 1 


19.05 




10 14 


1 1 


933 


1 c 1 7 




11571 


0 


10557 


.00 




TOTAL 


26 


11543 


.24 


4 










3 






REQUEST 31 


2 


11571 


0 


11571 


.00 


1 


TOTAL 


0 


11571 


.00 


0 






REQUEST 32 




4 


1 


3 


25.00 




73 


5 


64 


7 0 25 


3 


765 


2 


690 


.29 


2 


11571 


0 


1 0606 


.00 


1 


TOTAL 


6 


11563 


.07 


0 






REQUEST 33 




3 


3 


0 


100.00 




63 


a 


72 


10.00 


5 


1 1 36 


1 


1054 


.09 


4 


11571 


0 


1 0433 


.00 


3 


TOTAL 


1 2 


11559 


. 1 0 


2 

1 






REQUEST 34 


1 

0 


2 


1 


1 


50.00 




1 3 


1 


1 0 


9.09 




1 02 


3 


66 


3.37 




522 


2 


4 1 8 


.46 


7 


2950 


0 


2426 


.00 


6 


11571 


0 


862 1 


.00 


5 


TOTAL 


7 


11564 


.06 


4 










3 






REQUEST 35 


2 


1 


1 


0 


100.00 


1 


5 


2 


2 


50.00 


0 


47 


0 


42 


.00 




34 1 


3 


29 1 


1.02 




164 1 


4 


1296 


.3 1 




5477 


1 


3835 


.03 


2 


11571 


0 


6094 


.00 


1 


TOTAL 


1 1 


11560 


.10 


0 






REQUEST 36 




3 


3 


0 


100.00 




15 


4 


8 . 


33.33 


4 


105 


6 


84 


6.67 


3 


132 1 


0 


1216 


.00 


2 


1157 1 


0 


1 0250 


.00 


1 


total 


1 3 


1 1558 


.1 1 


0 



5 1 

33 2 

104 2 

39 1 3 

1 640 0 

504 6 0 

11571 0 

TOTAL a 

1 0 

1 2 2 

70 2 

264 1 

1 006 1 

4 364 3 

11571 0 

TOTAL 9 

2 2 

44 14 

339 1 4 

22 16 0 

11571 0 

TOTAL 30 

24 1 

37 1 6 

2617 1 

11571 0 

TOTAL 6 

1 1 

9 6 

27 6 

166 8 

1 504 4 

11571 0 

TOTAL 25 

1 1 

2 0 

7 2 

1 5 2 

56 6 

362 1 

2329 0 

11571 0 

TOTAL 12 

32 2 

650 , 3 

11571 0 

TOTAL 5 

1 2 a 

33 7 

197 12 

1 0 4 7 2 

11571 0 

TOTAL 29 



REQUEST 37 



4 


o 

o 

. 

o 

f\J 


26 


7.14 


69 


2.52 


284 


1 .05 


1249 


.00 


3406 


.00 


6525 


.00 


1 1563 


.07 


REQUEST 38 


1 


.00 


9 


18.16 


56 


3.45 


193 


.52 


743 


. 1 3 


3373 


.09 


7 1 87 


.00 


11562 


.08 


REQUEST 39 


0 


100.00 


28 


33.33 


28 1 


4.75 


1 879 


.00 


9353 


.00 


11541 


.26 


REQUEST 40 


23 


4.17 


34 1 


1.73 


2445 


.04 


8754 


.00 


1 1 563 


.07 


REQUEST 41 


0 


100.00 


2 


75.00 


1 2 


33.33 


1 33 


5.67 


1 332 


.30 


1 0067 


.00 


11546 


.22 


REQUEST 42 


0 


100.00 


1 


.00 


3 


40.00 


6 


25.0^ 


35 


14.6^ 


325 


. 3 1 


1947 


.00 


9242 


.00 


1 1 559 


. 1 0 


REQUEST 43 


30 


6.25 


815 


.37 


1 072 1 


.00 


1 1566 


.04 


REQUEST 44 


4 


6 6.67 


1 4 


33.33 


152 


7.32 


848 


.24 


1 0524 


.00 


1 1542 


.25 



0 

0 

3 

2 

1 

0 

4 

3 

2 

i 

0 

4 

3 

2 

1 

0 

3 

2 

1 

0 

3 

2 

1 

0 

5 

4 

3 

2 

1 

0 

4 

3 

2 

1 

0 

4 

3 

*3 

c 







request 45 


1157 1 


0 


11571 


.00 


total 


0 


11571 


.00 






request 46 


11571 


0 


11571 


.00 


total 


0 


11571 


.00 






REQUEST 47 


2 


1 


1 


50.00 




4 I 


43 


48.81 


796 


32 


678 


4.5 1 


11571 


1 0 


10765 


.09 


total 


84 


1 1 487 


.73 






request 48 


1 0 


10 


0 


lOO.OO 


65 


1 6 


39 


29.09 


232 


7 


2 1 0 


3 . 23 


175 1 


3 


1466 


.20 


11571 


0 


9820 


0 00 


TOTAL 


3 6 


1 1 535 


.31 






request 49 


3 


3 


0 


100.00 


34 


1 


30 


3.23 


3 1 7 


3 


280 


1.06 


1 134 


0 


817 


.00 


11571 


0 


10437 


.00 


TOTAL 


7 


11564 


.06 






request 50 


4 


3 


1 


. 

o 

o 


165 


3 


1 58 


1.86 


772 


2 


605 


.33 


11571 


0 


1 0799 


o 

o 


TOTAL 


8 


1 1563 


.07 






request 51 


9 


1 


8 


11.11 


163 


2 


152 


1.30 


1683 


2 


1518 


. 1 3 


11571 


0 


9688 


.00 


TOTAL 


5 


1 1566 


.04 






REQUEST 52 


1 


0 


1 


.00 


8 


5 


2 


7 1.43 


37 


9 


20 


3 1.03 


240 


27 


1 76 


13.30 


1 44 1 


7 


1 1 94 


.58 


11571 


0 


10130 


.00 


TOTAL 


48 


115 2 3 


.4 1 






request 53 


20 


8 


1 2 


40.00 


158 


2 1 


1 17 


15.22 


750 


3 


589 


.5 1 


3 1 60 


0 


2410 


.00 


11571 


0 


841 1 


• 00 


total 


32 


1 1 539 


.28 






REQUEST 54 


3 


1 


2. 


3 3 . 3,3 


38 


2 


33 


5 .71 


375 


0 


337 


.00 


2578 


0 


2203 


.00 


11571 


0 


6993 


.00 


TOTAL 


3 


1 1568 


.03 









request 55 


4 


5 


3 


2 


60.00 


3 


39 


1 


33 


2.94 


2 


583 


1 


543 


. 18 


I 


3 137 


2 


2552 


• 08 


0 


11571 


0 


8434 


o 

o 

. 




TOTAL 


7 


1 1 564 


.06 








REQUEST 56 


3 


7 


0 


7 


• 00 


2 


2 1 4 


1 


206 


• 48 


1 


1848 


0 


1634 


• 00 


0 


11571 


0 


9723 


.00 




TOTAL 


1 


11570 


• 01 








REQUEST 57 


5 


4 


2 


2 


50.00 


4 


35 


3 


28 


9.68 


3 


195 


7 


153 


4.38 


2 


1135 


4 


936 


• 43 


1 


4374 


0 


3239 


• 00 


0 


11571 


0 


7 197 


• 00 




TOTAL 


1 6 


1 1 555 


• 1 4 








REQUEST 58 


5 


1 


0 


1 


.00 


4 


1 3 


4 


8 


33.33 


3 


73 


10 


50 


16.67 


2 


572 


6 


493 


1 .20 


1 


4038 


5 


346 1 


0 1 4 


0 


11571 


0 


7533 


.00 




TOTAL 


25 


11546 


.22 



REQUEST 59 

5101 .00 



4 


6 


1 


4 


20.00 


3 


1 1 7 


1 3 


98 


11.71 


2 


1 09 1 


3 


97 1 


.31 


1 


4565 


0 


3474 


.00 


0 


11571 


0 


7006 


• 00 




TOTAL 


1 7 


11554 


. 1 5 








REQUEST 60 


3 


8 


1 


7 


12.50 


2 


2 1 0 


8 


1 94 


3.96 


1 


2 118 


1 


1 907 


.05 


0 


11571 


0 


9453 


• 00 




TOTAL 


1 0 


11561 


.09 








REQUEST 61 


4 


1 


0 


1 


• 00 


3 


53 


1 0 


42 


19.23 


2 


389 


2 


334 


• 6 0 


1 


2456 


1 


2066 


• 05 


0 


11571 


0 


9 115 


• 00 




total 


13 


11558 


• 1 1 








REQUEST 62 


3 


34 


22 


1 2 


64.7 1 


2 


399 


12' 


353 


3.29 


1 


3151 


5 


2747 


. 18 


0 


11571 


0 


8420 


.00 




TOTAL 


39 


11532 


.34 



132 









REQUEST 63 








REQUEST 72 


4 


1 8 


0 


18 


.00 


4 


2 


1 


1 


50.00 


3 


68 


6 


44 


12.00 


3 


1 8 


1 


1 5 


6.25 


2 


272 


4 


200 


1 .96 


2 


23 1 


1 


2 1 ? 


.47 


1 


1814 


0 


1542 


.00 


1 


260 1 


1 


2369 


.04 


0 


1157 1 


0 


9757 


.00 


0 


11571 


0 


6970 


.00 




TOTAL 


10 


11561 


.09 




TOTAL 


4 


11567 


.03 








REQUEST 64 








REQUEST 73 


4 


4 


1 


3 


25.00 


4 


1 


0 


1 


.00 


3 


74 


2 


68 


2.86 


3 


1 9 


4 


1 4 


22.22 


2 


539 


7 


458 


1.51 


2 


260 


4 


237 


1.66 


1 


3368 


0 


2849 


.00 


1 


1496 


0 


1236 


.00 


0 


11571 


0 


8 183 


.00 


0 


11571 


0 


1 0075 


.00 




TOTAL 


10 


11561 


.09 




TOTAL 


8 


11563 


.07 








REQUEST 6b 








REQUEST 74 


4 


6 


0 


6 


.00 


5 


1 


0 


1 


.00 


3 


50 


0 


44 


.00 


4 


25 


4 


20 


16.67 


2 


502 


1 


45 1 


.22 


3 


25 1 


1 2 


2 1 4 


5.3 1 


1 


3009 


0 


2507 


.00 


2 


1115 


1 1 


853 


1 .27 


0 


1157 1 


0 


8562 


.00 


1 


3663 


0 


2548 


.00 




TOTAL 


1 


11570 


. 0 1 


0 


11571 


0 


7908 


.00 














total 


27 


11544 


.23 








REQUEST 66 












0 


1157 1 


0 


11571 


.00 








REQUEST 75 




TOTAL 


0 


11571 


.00 


4 


1 


1 


0 


100.00 












3 


23 


5 


1 7 


22.73 








REQUEST 67 


2 


297 


1 5 


259 


5.47 


6 


3 


0 


3 


.00 


1 


2493 


1 2 


2 184 


.55 


5 


1 4 


0 


1 1 


.00 


0 


11571 


0 


9078 


.00 


4 


80 


1 


65 


1 .52 




TOTAL 


33 


11538 


• 29 


3 


4 1 5 


2 


33 3 


.60 












2 


1633 


0 


12 18 


.00 








REQUEST 76 


1 


4696 


0 


3063 


.00 


4 


1 


1 


0 


100.00- 


0 


11571 


0 


6875 


.00 


3 


18 


7 


1 0 


4 1.10 




TOTAL 


3 


11568 


.03 


2 


282 


5 


259 


1 .09 












1 


2532 


1 


2249 


.04 








REQUEST 68 


0 


11571 


0 


9039 


.00 


4 


1 1 


5 


6 


45.45 




TOTAL 


1 4 


1 1557 


. 1 2 


3 


123 


26 


86 


23.2 1 












2 


81 1 


1 3 


675 


1 .89 








REQUEST 77 


1 


36 1 4 


0 


2803 


.00 


4 


2 


1 


1 


50.00 


0 


11571 


0 


7957 


.00 


3 


40 


3 


35 


7.09 




TOTAL 


44 


11527 


.38 


2 


57 3 


1 


532 


. 19 












1 


3039 


0 


2466 


.00 








REQUEST 69 


0 


11571 


0 


8532 


.00 


3 


4 


3 


1 


75.00 




TOTAL 


5 


11566 


.04 


2 


39 


1 5 


20 


42.86 












1 


845 


3 


803 


.37 








REQUEST 76 


0 


1157 1 


0 


1 0726 


.00 


3 


8 


3 


5 


37.50 




TOTAL 


2 1 


11550 


. 18 


2 


1 99 


1 4 


177 


7.33 












1 


1874 


4 


167 1 


.24 








REQUEST 70 


0 


11571 


0 


9697 


.00 


4 


2 


2 


0 


1 00. OG 




total 


2 1 


11550 


. 1 8 


3 


75 


1 7 


56 


23.29 








REQUEST 79 


2 


2 1 0 


6 


129 


4.44 


3 


1 8 


1 7 


1 


94.44 


1 


1375 


5 


1160 


.43 


2 


165 


1 5 


1 32 


10.20 


0 


1157 1 


0 


10196 


4* J 0 


1 


1773 


0 


1608 


.00 




TOTAL 


30 


11541 


.26 


0 


11571 


0 


9790 


• 00 








REQUEST 71 




TOTAL 


32 


11539 


.28 


4 


25 


0 


25 


.00 












3 


209 


1 7 


167 


9.24 








REQUEST 60 


2 


1022 


5 


808 


.62 


•*:* 2 


45 


13 


32 


20.09 


1 


4036 


0 


30 1 4 


.00 


1 


1 064 


5 


10 14 


.49 




1157 1 


0 


7535 


.00 


0 


11571 


0 


1 0507 


.00 


ERIC 


TOTAL 


22 


1 1549 


.19 


133 


TOTAL 


18 


1 1553 


• 1 6 



3 

2 

1 

0 

3 

2 

1 

0 

3 

2 

1 

0 

3 

2 

1 

0 

6 

5 

4 

3 

2 

1 

0 







REQUEST 81 


3 


3 


0 


100.00 


220 


1 7 


200 


7 .83 


1564 


1 1 


1353 


.81 


11571 


1 


9986 


. 0 1 


TOTAL 


32 


1 1539 


.28 






REQUEST 82 


1 


1 


0 


100.00 


94 


62 


31 


66.67 


1 054 


3 


9 5:’ 


. 3 1 


11571 


0 


10517 


.00 


TOTAL 


66 


1 1505 


.57 






REQUEST 83 


1 9 


6 


1 3 


3 1.56 


464 


2 


463 


» 43 


24 1 7 


1 


1932 


.05 


11571 


0 


9 154 


.00 


TOTAL 


Q 


1 1562 


.08 






REQUEST 84 


24 


4 


20 


16.67 


1 1 5 


1 8 


73 


19.78 


1 1 27 


2 


10 10 


.20 


11571 


0 


10444 


.00 


TOTAL 


24 


1 1547 


.2 1 






REQUEST 85 


3 


0 


3 


.00 


22 


5 


1 4 


26.32 


8 1 


9 


50 


15.25 


22 1 


8 


1 32 


5.7 1 


607 


3 


383 


.78 


269 1 


0 


2084 


.00 


1157 1 


0 


6880 


.00 


TOTAL 


25 


1 1546 


.22 









REQUEST 86 


4 


8 


1 


7 


12.50 


3 


1 03 


1 9 


76 


20.00 


2 


808 


9 


696 


1 .28 


1 


374 1 


0 


2933 


.00 


0 


11571 


0 


7830 


.00 




TOTAL 


29 


1 1542 


.25 








REQUEST 87 


2 


1 1 


4 


7 


3 6.36 


1 


464 


8 


445 


1.77 


0 


11571 


I 


11106 


.0 1 




TOTAL 


1 3 


1 1 558 


. 1 1. 








REQUEST 88 


4 


5 


0 


5 


.00 


3 


3 1 


1 


25 


3.85 


2 


333 


3 


299 


.99 


1 


2786 


0 


2 453 


.00 


0 


11571 


0 


8785 


.00 




TOTAL 


4 


1 1567 


.03 








REQUEST 89 


4 


3 


1 


2 


33.33 


3 


57 


9 


45 


16.67 


2 


393 


1 2 


324 


3.57 


1 


2812 


0 


2419 


.00 


0 


11571 


0 


8759 


.00 




TOTAL 


22 


1 1 549 


. 1 9 








REQUEST 90 


5 


8 


1 


7 


12.50 


4 


62 


1 2 


42 


22.22 


3 


244 


1 0 


172 


5.49 


2 


886 


2 


640 


.31 


1 


3364 


1 


2 477 


.04 


0 


11571 


0 


8207 


.00 




TOTAL 


26 


1 1545 


.22 



134 



o in 



4 

3 

2 

1 

0 



3 

2 

1 

0 



4 

3 

2 

1 

0 



3 

2 

1 

0 



4 

3 

2 

1 

0 







REQUEST 91 








REQUEST 96 


4 


0 


4 


• 00 


3 


1 3 


0 


1 3 


• 00 


37 


9 


24 


27*27 


2 


1 49 


1 2 


1 24 


6.82 


282 


9 


236 


3*67 


1 


1 758 


0 


1 609 


• 00 


2078 


5 


179 1 


• 28 


0 


11571 


0 


96 13 


• 00 


11571 


0 


9493 


• 00 




TOTAL 


1 2 


11559 


• 1 0 


TOTAL 


23 


1 1548 


• 20 


























REQUEST 97 






REQUEST 92 


4 


1 


0 


1 


.00 


1 2 


2 


1 0 


16.67 


3 


1 7 


5 


1 1 


3 1.25 


23 1 


4 


2 1 5 


I .83 


2 


453 


1 8 


4 1 8 


4.13 


2076 


2 


1643 


• 1 1 


1 


2 1 07 


4 


1650 


.24 


11571 


0 


9495 


• 00 


0 


.1571 


2 


9462 


.02 


TOTAL 


8 


11563 


.07 




TOTAL 


29 


11542 


.25 






REQUEST 93 








REQUEST 96 


2 


1 


1 


50.00 


4 


6 


2 


4 


33.33 


1 3 


2 


9 


18.18 


3 


29 


6 


1 7 


26.09 


47 


2 


32 


5.88 


2 


165 


26 


1 10 


19.12 


2 1 7 


1 3 


157 


7.65 


1 


7 56 


1 


590 


. 1 7 


856 


2 


637 


. 3 1 


0 


11571 


0 


10815 


>00 


3362 


0 


2506 


.00 




TOT AL 


35 


1 1 536 


.30 


11571 


0 


8209 


.00 












TOTAL 


20 


11551 


. 1 7 








REQUEST 99 










6 


5 


1 


4 


20.00 






REQUEST 94 


5 


33 


3 


25 


10.7 1 


1 3 


3 


1 0 


23.08 


4 


1 1 7 


5 


79 


5.95 


2 1 2 


1 


198 


.50 


3 


408 


5 


286 


1.72 


2206 


3 


1 99 1 


. 1 5 


2 


1312 


0 


904 


.00 


11571 


0 


9365 


.00 


1 


356 1 


0 


2249 


• 00 


TOTAL 


7 


1 1 564 


• 06 


0 


11571 


0 


80 1 0 


• 00 












TOTAL 


1 4 


1 1557 


• 1 2 






REQUEST 95 












1 


1 


0 


100.00 












8 


3 


4 


42.86 












68 


2 


58 


3.33 












524 


3 


453 


• 6 6 












2659 


1 


2 1 34 


.05 












11571 


0 


89 12 


• 00 












TOTAL 


1 0 


11561 


.09 
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APPENDIX B8 



Performance of Typical Strategies for Varying Cut-off K 



See VIII. 2, 3, 4, 5 for explanation of the tables. 

S. D. = Standard Deviation of the percentages averaged 
in the immediately preceding columns. 



93 REQUESTS, RUN 13(KWS). TABLE RPK(93, 13). 



Cut-o.ff 

K 


Average 

Output 

K ' 


kVk 


Rele- 

vant 


Irrel- 

evant 


Not 

Assessed 


Output 


0 


.00 


.000 


0 


0 


0 


0 


1 


.22 


.220 


9 


1 1 


0 


20 


2 


.94 


.470 


49 


38 


0 


87 


3 


1 .60 


.533 


7 1 


78 


0 


149 


4 


2.05 


.5 13 


83 


108 


0 


191 


5 


2.74 


• 548 


102 


153 


0 


255 


6 


3.43 


.572 


132 


187 


0 


319 


7 


4.35 


• 62 1 


1 54 


25 1 


0 


405 


8 


5.14 


• 643 


179 


299 


0 


^78 


9 


5.58 


.620 


191 ^ 


328 


0 


519 


10 


6.86 


.666 


246 


392 


0 


638 


1 1 


7.96 


.724 


280 


460 


0 


740 


12 


6.04 


.670 


282 


466 


0 


748 


13 


9.47 


.728 


2^7 


584 


0 


831 


14 


9.73 


.695 


301 


604 


0 


905 


15 


9.95 


• 663 


3 1 2 


613 


0 


925 


1 6 


10.42 


• 65 1 


324 


645 


0 


969 


17 


11.05 


• 650 


334 


694 


0 


1028 


18 


1 1 o 92 


• 662 


37 1 


738 


0 


1 109 


19 


13.27 


.698 


392 


842 


0 


1234 


20 


14 o 20 


.7 10 


400 


921 


0 


132 1 


2 1 


15.18 


.723 


42 1 


991 


0 


1412 


22 


15.97 


.726 


439 


1046 


0 


1485 


23 


17.35 


.754 


469 


1 145 


0 


16 14 


24 


17.81 


.742 


483 


1 173 


0 


1 656 


25 


18.22 


.729 


489 


1205 


0 


1694 


26 


18.22 


.701 


489 


1205 


0 


1694 


27 


19.20 


.71 1 


489 


1297 


0 


1786 


28 


20.29 


.725 


528 


1359 


0 


1 887 


29 


21.61 


.745 


550 


1460 


0 


2010 


30 


22.17 


.739 


565 


1497 


0 


2062 


3 1 


23.12 


.746 


576 


1574 


0 


2150 


32 


23.12 


.723 


576 


1574 


0 


2150 


33 


23.12 


.701 


576 


1574 


0 


2150 


34 


23.12 


• 6 8 0 


576 


1574 


0 


2150 


35 


23.12 


• 661 


576 


1574 


0 


2150 


36 


24 . 4 


.671 


594 


1651 


0 


2245 


37 


24.14 


.652 


594 


1651 


0 


2245 


38 


24.73 


.65 I 


610 


1690 


0 


2300 


39 


27.58 


.707 


646 


1919 


0 


2565 


40 


28.99 


.725 


648 


2046 


0 


2696 


41 


28.99 


'v707 . , 


648 


2046 


0 


2696 


42 


29.61 


.705 • 


650 


2104 


0 


2754 


43 


30.20 


.702 


671 


2136 


0 


2609 


44 


32 o 25 


.733 


695 


2304 


0 


2999 



K 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

1 0 

1 1 

1 2 

1 3 

1 4 

1 5 

1 6 

1 7 

1 8 

1 9 

20 

2 1 

22 

23 

24 

25 

26 

27 

28 

29 

30 

3 1 

32 

33 

34 

35 

36 

3T 

38 

39 

40 

4 1 

42 

43 

44 



Average 


Overall 


Overall 


Average 


Average 


S.D. 


S.D. 


Output 


^Prec- 




^Prec- 


^Known 


%Prec- 


XKnovm 


k' 


ision 


Recall 


ision 


Recall 


ision 


Recall 


.00 


.00 


.00 


.00 


.00 


• 00 


,00 


.22 


45.00 


.43 


9.68 


.55 


29,57 


1,90 


.94 


56.32 


2.34 


25.61 


3.93 


3V ,75 


8,90 


1 .60 


47.65 


3.40 


3 1.34 


5.94 


39,7 1 


10,63 


2.05 


4 3.46 


3.97 


^^.53 


6.88 


38,87 


11,36 


2.74 


40,00 


4.88 


33.5 1 


8.15 


37,7 1 


11,96 


3.43 


4 1.38 


6.32 


34,87 


10.40 


36,63 


15,47 


4.35 


36.02 


7.37 


36.77 


11.79 


36,00 


15,80 


5.14 


37.45 


8 .56 


37.66 


12.91 


35,30 


15,78 


5.58 


3 6.80 


9.14 


37.76 


13.14 


3 4,64 


15,73 


6.86 


38.56 


11.77 


39.16 


16.10 


33,37 


18,37 


7.96 


37.84 


13.40 


39.09 


16.04 


37.25 


18,89 


8.04 


37.70 


13.49 


38 .99 


16.22 


32,25 


19,03 


9.47 


33.7 1 


14.21 


38. 35 


16.91 


3 1,40 


16,66 


9.73 


33.26 


M . 40 


38.52 


19.07 


31,23 


18,58 


9.95 


33.73 


14.93 


38.57 


19.37 


3 1,25 


18,5^- 


10.42 


33.44 


15.50 


38.15 


20.04 


3 1,23 


18,60 


11.05 


32.49 


15.98 


37.54 


20.64 


30, 48 


18,98 


11.92 


33.45 


\ 7.75 


38.15 


2 1.76 


30,37 


19,05 


13.27 


3 1.77 


18.76 


36.2 0 


22.64 


28,72 


19,25 


14.20 


30.28 


19.14 


35.6 1 


23.5 4 


28,92 


19,20 


15.18 


29.82 


20.14 


35.68 


25.00 


28 V 85 


20,62 


15.97 


2 9 . 5 o 


2 1.00 


34.94 


26.4 1 


28,66 


2 2,22 


17.35 


29.06 


22.44 


34.22 


27.8 1 


28,11 


22,77 


17.61 


29.17 


23.11 


33.54 


26.3 1 


27,26 


22.6 1 


18.22 


28.87 


23.40 


33.69 


26.60 


27,12 


22,69 


18.22 


28.87 


23.40 


33.69 


26.60 


2 7,12 


22,69 


19.20 


27.38 


23.40 


33.11 


26.80 


27,12 


22,69 


20.29 


27.98 


25.26 


33. 58 


30.19 


27,06 


22,97 


2 1.61 


27.36 


26.32 


32.96 


30.86 


26.87 


23, 1 4 


22.17 


27.40 


27.03 


32.24 


31.27 


25,95 


23,06 


23.12 


26.79 


27.56 


3 1.94 


31.62 


26,07 


22,69 


2 3.12 


26.79 


27.56 


31.94 


31.62 


26,07 


22,69 


2 3.12 


26.79 


27.56 


3 1.94 


31.62 


26,07 


22,69 


23. 1 2 


26.79 


27.56 


31.94 


3 1.62 


26,07 


22,69 


23. 1 2 


26.79 


27.56 


3 1.94 


3 1.62 


26,07 


22,69 


24.14 


26.46 


28.42 


3 1.66 


32.85 


26,04 


23,56 


24.14 


26.46 


26.42 


3 1.66 


32.85 


2 6,04 


23,56 


24.73 


26.52 


29.19 


3 1.23 


33.33 


25o07 


23,90 


27.58 


25.19 


30.9 1 


29.30 


35.29 


23,76 


24,42 


28.99 


24 .04 


31.00 


28.97 


35.50 


24,00 


24,28 


28.99 


24.04 


3 1.00 


28.97 


35.50 


24.00 


24,28 


29.6 1 


23.60 


3 1.10 


28.85 


35.74 


24,06 


24,26 


30.20 


23.89 


32.11 


26.6 1 


36.19 


23.62 


24,35 


32.25 


23.17 


33.25 


27.65 


37.98 


22.59 


2 4,77 



Cut-off Average 



K 


Output 

K' 


45 


33.15 


46 


33.15 


47 


33.15 


48 


34.66 


49 


34.66 


50 


34.66 


51 


35.47 


52 


36.11 


53 


36.11 


54 


36.11 


55 


37.55 


56 


38.57 


57 


36.57 


58 


40.46 


59 


40.46 


60 


40.46 


61 


42.44 


62 


43.63 


63 


43.63 


64 


43.63 


65 


43.63 


66 


43.63 


67 


44.90 


66 


46.11 


69 


47.66 


70 


49.76 


71 


49.76 


72 


50.66 



O 




Rele- 

vant 


Irrel- 

evant 


Not 

Assessed 


Output 


736 


2347 


0 


3083 


736 


2347 


0 


3083 


736 


2347 


0 


3083 


799 


2443 


0 


3242 


799 


2443 


0 


3242 


799 


2443 


0 


3242 


600 


2499 


0 


3299 


609 


2549 


0 


3358 


609 


2549 


0 


3358 


609 


2549 


0 


3358 


645 


2647 


0 


3492 


664 


2723 


0 


3587 


864 


2723 


0 


3587 


697 


2666 


0 


3763 


897 


2666 


0 


3763 


897 


2666 


0 


3763 


913 


3034 


0 


3947 


926 


3 132 


0 


4058 


926 


3 132 


0 


4058 


926 


3132 


0 


4058 


926 


3 1 32 


0 


4058 


926 


3132 


0 


4058 


928 


3248 


0 


4176 


954 


3334 


0 


4288 


961 


347 1 


0 


4432 


991 


3637 


0 


4628 


99 1 


3637 


0 


4628 


1035 


3678 


0 


4713 



kVk 

• 737 

.72 1 

.705 

• 726 

.7 11 

• 697 

.695 

.694 

.63 1 

• 669 

.663 

.669 

.677 

• 696 

• 666 

.674 

.696 

.704 

• 693 

• 662 

• 67 1 

• 66 1 

.670 

.676 

.691 

• 71 1 

.701 

• 704 



138 



Cut-off 

K 


Average 

Output 


Overall 

VoPrec- 

ision 


Overall 

%Known 

Recall 


Ave rage 
%Prec- 
ision 


Average 
% Known 
Re call 


S.D. 

tPre c- 
islon 


S-D, 

7oKnown 

Recall 


45 


33.15 


23.87 


35.22 


27.64 


38.5 1 


22.56 


24 . 50 ' 


46 


33.15 


23.87 


35.22 


27.64 


36.5 1 


22.56 


24.50 


47 


33.15 


23.87 


35.22 


27.64 


36.5 1 


22.56 


24.50 


48 


34.66 


24.65 


38.23 


27.30 


39.66 


2 1.66 


24.55 


49 


34.86 


24.65 


38.23 


27.30 


39.66 


2 1.66 


24.55 


50 


34.86 


24.65 


38.23 


27.30 


39.66 


21.66 


24.55 


51 


35.47 


24.25 


38.28 


27.28 


40.4 1 


2 1.69 


2 5.31 


52 


36.11 


24.09 


38.71 


27.22 


40.60 


21.71 


25.27 


53 


36.11 


24.09 


38.7 1 


27.22 


40.60 


2 1.71 


25.27 


54 


36.11 


24.09 


38.71 


27.22 


40.60 


2 1.71 


25.27 


55 


37.55 


24.20 


40.43 


26.85 


4 1.58 


2 1.46 


25.35 


56 


36.57 


24.09 


4 1.34 


2 6.92 


42.29 


21.42 


25.20 


57 


36.57 


24.09 


41 .34 


26.92 


42.29 


21.42 


25.20 


58 


40.46 


23.84 


42.92 


26.66 


43.45 


2 1.40 


25.46 


59 


40.46 


23.84 


42.92 


26.66 


43.45 


21.40 


25.46 


60 


40.46 


23.84 


42.92 


26.66 


43.45 


21.40 


25.46 


61 


42.44 


23.13 


43.66 


25.89 


44.28 


21.16 


26.02 


62 


43.63 


22.82 


44.3 1 


25.84 


45.10 


21.20 


26.00 


63 


4 3.63 


22.82 


44.3 1 


25.84 


45.10 


21.20 


26.00 


64 


4 3.63 


22.82 


44.31 


25.84 


45.10 


2 1.20 


26.00 


65 


43.63 


22.82 


44.3 1 


25.84 


45.10 


2 1.20 


26.00 


66 


43.63 


22.82 


44.3 1 


25.84 


45.10 


2 1.20 


26.00 


67 


44.90 


22.22 


44.40 


25.56 


45.53 


2 1.33 


26.24 


68 


4 6.11 


22.25 


45.65 


25.35 


46.17 


2 1.23 


26 . 1 2 


69 


47.66 


2 1.68 


45.96 


25.05 


46.60 


2 1.22 


26.16 


70 


49.76 


2 1.41 


47.42 


24.96 


47.59 


2 1.25 


26.15 


71 


49.76 


2 1.41 


47.42 


24.96 


47.59 


2 1.25 


26.15 


72 


50.66 


2 1.96 


49.52 


25.07 


46.22 


2 1.34 


26.06 
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93 REQUESTS, RUN 



Off 

K 


Average 

Output 


kVk 


0 


• 00 


.000 


1 


• 54 


.540 


2 


1 «4 1 


.705 


3 


2 *40 


• 800 


4 


3« 1 2 


.780 


5 


4.13 


• 826 


6 


5.06 


• 843 


7 


6. 1 1 


.87 3 


6 


6.66 


• 858 


9 


8.28 


.920 


10 


9.66 


• 968 


1 1 


10.80 


• 982 


1 2 


11.54 


• 962 


1 3 


12.55 


.965 


1 4 


13.13 


.938 


15 


13.63 . 


• 922 


1 6 


14.52 


.908 


17 


15.39 


.905 


1 8 


16.31 


• 906 


19 


17.49 


.92 1 


20 


18.63 


• 932 


2 1 


16.91 


.900 


22 


20.24 


• 920 


23 


2 1.54 


.937 


24 


22.68 


.945 


25 


23.8 1 


.952 


26 


24.96 


• 960 


27 


2 5.44 


.942 


26 


27.01 


.965 


29 


27.56 


.951 


30 


26.53 


.951 


3 1 


29.11 


.939 


32 


30.29 


.947 


33 


30.74 


• 932 


34 


31.49 


• 926 


35 


32.13 


.918 


36 


32.80 


• 911 


3 7 


34.20 


• 924 


3B 


35.05 


• 922 


39 


35.96 


• 922 


40 


37.47 


.937 


41 


38.42 


.937 


42 


39.95 


.951 


43 


4 1.42 


• 963 


44 


42.46 


.965 


45 


43.34 


• 963 


46 


44.19 


• 961 


47 


45.35 


• 965 


48 


45.99 


.958 


49 


46.94 


.958 


50 


48.37 


.967 


5 1 


49.31 


.967 


52 


50.32 


• 968 


53 


5 1.12 


• 965 


54 


52.12 


• 965 


55 


53.22 


• 968 


56 


53.77 


• 960 


57 


54.78 


• 961 


58 


55.56 


.958 


59 


56.45 


.957 


60 


58.76 


• 979 


6 1 


59.58 


.977 


62 


60.29 


.972 


63 


62.40 


.990 


64 


63.45 


.99 1 


65 


64.67 


.995 


66 


65.1 0 


• 986 


67 


66.10 


.987 


68 


66.42 


.977 


69 


6 7.65 


. 9 8‘0 


70 


68.73 


.98 2 


71 


70.08 


.987 


72 


70.7 1 


.982 



TABLE RPK(93,28). 



Irrel- 


Not 


Output 


evant 


Assessed 




0 


0 


0 


32 


0 


50 


7 1 


0 


131 


129 


0 


223 


176 


0 


290 


248 


0 


364 


307 


0 


471 


383 


0 


568 


432 


0 


638 


536 


0 


770 


622 


0 


900 


706 


0 


1004 


767 


0 


1073 


834 


0 


1 1 67 


077 


0 


122 1 


920 


0 


1286 


973 


0 


1350 


1037 


0 


1431 


1085 


0 


1517 


1 1 55 


0 


1627 


1235 


0 


1733 


1260 


0 


1759 


1356 


0 


1862 


1462 


0 


2003 


1560 


0 


2 109 


1652 


0 


22 14 


1735 


0 


232 1 


1769 


0 


2366 


1904 


0 


2512 


1947 


0 


2565 


2028 


0 


2653 


2078 


0 


2707 


2 160 


0 


2817 


2 1 93 


0 


2059 


2252 


0 


2929 


2301 


0 


2988 


2358 


0 


3050 


2467 


0 


3181 


2519 


0 


3260 


2597 


0 


3344 


2669 


0 


3485 


273 1 


0 


3573 


2855 


0 


3715 


297 1 


7 


3852 


3065 


7 


3949 


3128 


7 


4031 


3195 


7 


4 110 


3271 


30 


42 18 


3327 


30 


4277 


3405 


30 


4365 


3529 


30 


4498 


3601 


36 


4586 


3653 


36 


4660 


3706 


52 


4754 


3794 


52 


4847 


3878 


52 


4949 


3924 


52 


5001 


4012 


56 


5095 


4073 


58 


5167 


4 1 50 


60 


5250 


4348 


60 


5 4 65 


4420 


60 


554 1 


447 1 


6 1 


5607 


4638 


63 


5803 


47 1 7 


74 


5901 


4816 


86 


60 1 4 


4854 


86 


6054 


4933 


86 


6 147 


4957 


89 


6177 


5025 


129 


629 1 


5107 


139 


6392 


5187 


167 


6517 


5236 


172 


6576 



28( i3Wl4) 

Rele- 

vant 

0 

IB 

60 

94 

1 14 

136 

164 

105 

206 

234 

27B 

290 

306 

333 

344 

366 

377 

394 

432 

472 

498 

499 

526 

54 1 

549 

562 

586 

597 

608 

6 1 8 

625 

629 

657 

666 

677 

687 

692 

714 

74 1 

747 

816 

642 

860 

874 

877 

896 

908 

9 17 

920 

930 

939 

949 

99 1 

996 

1001 

1019 

1025 

1027 

1036 

1040 

1057 

1061 

1075 

1 102 

1110 

1112 

1114 

1 128 

1131 

1 137 

1 146 

1 163 

1 1 68 



off 


Average 


Overall 


O^ve rail 


Average 


Ave rage 


S.D. 


S.D. 


K 


Output 

. 


%Prec- 

ision 


/iKnown 

Recall 


7oPrec- 

islon 


% Known 
Recall 


7oPrec- 

ision 


7o Known 
Re ca 1 1 


0 


• 00 


• 00 


• 00 


• 00 


.00 


• 00 


• 00 


1 


• 5^4 


36.00 


* 8 6 


19.35 


1 .02 


39.51 


2.61 


2 


1 * 4 "! 


45.80 


2.87 


33.33 


4.32 


4 1.04 


7.52 


3 


2*40 


42.15 


4.50 


37.22 


6.92 


36.92 


9.8 1 


4 


3*12 


39.3 1 


5.45 


38.23 


8.15 


35.01 


10.74 


5 


4 * 13 


35.42 


6*51 


36.35 


9.58 


32.89 


11.71 


6 


5*06 


34.62 


7.85 


37.10 


12*33 


31.11 


15.43 


7 


6. 1 1 


32.57 


8*85 


33.33 


13*28 


27.65 


15.72 


8 


6*86 


32.29 


9.86 


33.09 


14*47 


27.64 


16.25 


V 


8.28 


30.39 


11*20 


31.04 


15*96 


25.73 


16.63 


10 


9*68 


30.89 


13*30 


32.24 


18*90 


26.29 


19.16 


•i 1 


10.80 


29.68 


14*26 


31.62 


20*1 0 


25.55 


19.22 


1 2 


11.54 


28.52 


14*64 


30.97 


20*9 1 


24.66 


19.04 


13 


12.55 


28.53 


15*93 


30.22 


22*33 


23.15 


19.42 


1 4 


13.13 


28.17 


16*46 


29.86 


22*75 . 


23.3 1 


19.43 


15 


13.83 


28.46 


17*51 


29.48 


23*73 


22.88 


19.79 


1 6 


14.52 


27.93 


18*04 


29.38 


24*51 


22.80 


19.92 


17 


15.39 


27.53 


18*85 


28.89 


25*20 


22.10 


19.86 


16 


16.31 


28.46 


20*67 


28.90 


26*04 


22.53 


19.94 


19 


17.49 


29.01 


22*58 


28.92 


27*46 


22.45 


19.97 


20 


16.63 


28.74 


23*83 


28.91 


28*68 


22.15 


19.52 


2 1 


16.91 


28.37 


23*88 


28.59 


28*75 


2 1.96 


19.50 


22 


20.24 


27.95 


25*17 


28.00 


29*89 


2 1.73 


20.17 


23 


2 1.54 


27.01 


25*89 


27.41 


30*72 


2 1.57 


20.00 


24 


22.66 


26.03 


26*27 


26.74 


3 1*04 


2 1.61 


19.99 


25 


23.81 


25.36 


26*89 


26.22 


3 1*59 


2 1.73 


20.06 


26 


2 4 ..9 6 


25.25 


28.04 


26.08 


32*50 


2 1 .08 


19.73 


27 


25.44 


25.23 


28*56 


25.96 


32*89 


21.16 


19.87 


28 


27.01 


24.20 


29*09 


25.64 


33*53 


2 1.04 


20.75 


29 


27.58 


24.09 


29*57 


25.60 


33*88 


20.92 


20.57 


30 


26.53 


23.56 


29*90 


25.24 


34*35 


20.88 


20.85 


3 1 


29.11 


23.24 


30 * 10 


25.06 


34*53 


20.95 


20.83 


32 


30.29 


23.32 


31.44 


24.97 


36*02 


20.84 


2 1.47 


33 


30.74 


23.29 


3 1 .87 


24.92 


36*4 1 


20.8 1 


2 1*55 


34 


3 1.49 


23.11 


32.39 


24.79 


36*86 


20.73 


2 1*38 


35 


32.13 


22.99 


32.87 


24.79 


37*35 


20.82 


2 1*90 


36 


32.60 


22.69 


33.11 


24.54 


37*54 


20.83 


2 1*90 


37 


34.20 


22.45 


34.16 


23.47 


38.26 


19.15 


2 1*78 


38 


35.05 


22.73 


35.45 


23.56 


39.56 


19.05 


22*39 


39 


35.96 


22.34 


35.74 


23.03 


39.75 


18.53 


22*38 


40 


37.47 


23.41 


39.04 


23.33 


4 1^13 


19.20 


22*51 


4 1 


36.42 


23.57 


40.29 


23.45 


42.05 


19.2 1 


22*17 


42 


39.95 


23.15 


4 1.15 


23.22 


43.10 


19.25 


22*43 


43 


4 1.42 


22.73 


41 .82 


22.88 


43.98 


19.27 


22*96 


44 


42.46 


22.25 


4 1.96 


22.74 


44.07 


19.37 


22*94 


45 


43.34 


22.27 


42.87 


22.76 


44.76 


19.38 


22*99 


4 6 


44.19 


22.13 


43.44 


22.70 


45.23 


19.34 


22*75 


4 7 


4 5.35 


2 1.90 


43.88 


22.61 


45.99 


19.30 


22*48 


4 8 


45.99 


2 1*66 


44.02 


22.44 


46.16 


19.17 


22*45 


49 


46.94 


2 1*45 


44.50 


22.22 


46.66 


19.11 


22*59 


50 


48.37 


21 .02 


44.93 


2 1.68 


47.00 


18.70 


22*34 


5 1 


49.3 1 


20*66 


45.41 


21.38 


47.3 1 


18.4 1 


22*3 4 


52 


50.32 


21*34 


47.42 


2 1.67 


48.83 


18.84 


2 2*92 


53 


51.12 


21*16 


47.66 


21.55 


49.09 


18.78 


23*03 


5 4 


52.12 


20,66 


47.89 


21.37 


49.46 


18.80 


23*23 


55 


53.22 


20*61 


48.76 


20.7 1 


50.11 


17.26 


23*68 


56 


53.77 


20*71 


49.04 


20.58 


50.26 


17.19 


23*70 


57 


54.78 


20*36 


49.14 


20*48 


50.80 


17.09 


23*11 


58 


55.56 


20*26 


49.57 


20*47 


51*45 


17.11 


23*45 


59 


56.45 


20*04 


49.76 


20*34 


51*54 


17.14 


23*42 


6 0 


58.76 


19*56 


50.57 


19*97 


52*66 


17.00 


23*4 1 


6 1 


59.56 


19*36 


50.77 


19*74 


52*79 


16.94 


23.50 


62 


60.29 


19*36 


5 1.44 


19*81 


53*35 


17.02 


23.45 


6 3 


62.40 


19*20 


52.73 


19*12 


54*06 


15.63 


22.96 


64 


63.45 


19*05 


53.11 


18*99 


54*26 


15.58 


22.87 


6 5 


64.67 


18*76 


53.21 


18*78 


54*32 


15.48 


22.83 


6 6 


65.10 


18*67 


53.30 


18*69 


54.40 


15.44 


22.84 


67 


66.10 


18*61 


53.97 


18*65 


55.30 


15.29 


22.76 


6 8 


66.42 


18*56 


54.11 


18*63 


55 . 44 - 


15.28 


22.74 


69 


67.65 


18*45 


54.40 


18*54 


55.79 


15.32 


22.74 


70 


68.73 


18*33 


54.83 


18*49 


56.08 


15.37 


22.77 


7 1 


70.08 


18*31 


55.65 


^ 18*33 


5 6.58 


15.17 


22.67 


72 


70.7 1 


18*24 


55.89 ■ 


18*29 


56.74 


15.22 


22.68 
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93 REQUESTS, RUN l4(MCS0l). TABLE RPK(93»14). 



Cut-Off 


Average 


kVk 


Rele- 


Irrel- 


Not 


Output 


K 


Output 

K' 




vant 


evant 


Assessed 




0 


• 00 


• 000 


0 


0 


0 


0 


I 


• 40 


• 400 


1 1 


26 


0 


37 


2 


1 ^34 


• 670 


44 


8 1 


0 


1 25 


3 


2^23 


• 743 


74 


133 


0 


207 


4 


3^25 


• 813 


98 


204 


0 


302 


5 


4^15 


• 830 


105 


281 


0 


386 


6 


4^96 


• 827 


1 29 


332 


0 


46 1 


7 


5^65 


• 807 


145 


380 


0 


525 


8 


6^69 


• 836 


170 


452 


0 


622 


9 


7^62 


• 847 


189 


520 


0 


709 


10 


8^74 


• 874 


228 


585 


0 


8 1 3 


1 1 


9^39 


• 854 


237 


636 


0 


873 


12 


10^58 


• 882 


260 


724 


0 


984 


13 


1 1 • 1 8 


• 860 


266 


774 


0 


1040 


1 4 


12^05 


• 861 


281 


840 


0 


1121 


15 


13^34 


• 889 


296 


945 


0 


124 1 


16 


14^44 


• 903 


308 


1 035 


0 


1343 


17 


14^90 


• 876 


3 1 4 


1 0/2 


0 


1386 


18 


15^75 


• 875 


321 


1 144 


0 


1465 


19 


16^98 


• 894 


332 


1247 


0 


1 579 


20 


18^37 


• 9 19 


374 


1 334 


0 


1 708 


21 


19^15 


• 912 


383 


1 398 


0 


178 1 


22 


19^67 


• 894 


394 


1 435 


0 


1829 


23 


2 1 ^68 


• 943 


428 


1588 


0 


2016 


24 


22^42 


• 934 


450 


1 635 


0 


2085 


25 


22.62 


• 945 


457 


1 740 


0 


2 197 


26 


22.95 


• 921 


460 


1 767 


0 


2227 


27 


25. \b 


• 933 


487 


1 855 


0 


2342 


28 


26^48 


• 946 


504 


1 959 


0 


2463 


29 


27^24 


• 939 


5 I 3 


2020 


0 


2533 


30 


21 .92 


• 93 1 


527 


2070 


0 


2597 


31 


28^44 


• 917 


527 


2118 


0 


2645 


32 


28^70 


• 897 


531 


2 138 


0 


2669 


33 


29*69 


• 900 


546 


2215 


0 


276 1 


'34 


29^83 


• 877 


548 


2226 


0 


2774 


35 


30^03 


• 858 


549 


2244 


0 


2793 


36 


30^63 


• 85 1 


559 


2290 


0 


2849 


37 


32^28 


• 872 


579 


2423 


0 


3002 


38 


33^90 


• 892 


606 


2547 


0 


3153 


39 


35^31 


*905 


647 


2637 


0 


3284 


40 


36^73 


• 918 


708 


2708 


0 


34 1 6 


41 


37^49 


• 9 14 


7 1 3 


2774 


0 


3487 


42 


38^30 


• 912 


728 


2834 


0 


3562 


43 


39^74 


• 924 


753 


2943 


0 


3696 


44 


40^95 


• 93 1 


755 


3053 


0 


3808 


45 


4 1 ^47 


• 922 


760 


3097 


0 


3857 


46 


43^03 


• 935 


774 


3228 


0 


4002 


47 


43*6 1 


• 928 


791 


3265 


0 


4056 


48 


44^52 


*928 


799 


334 1 


0 


4 140 


49 


45# 15 


• 921 


812 


3387 


0 


4 199 


50 


46*30 


• 926 


829 


3477 


0 


4306 


51 


46*89 


• 919 


833 


3528 


0 


436 1 


52 


47*10 


• 906 


834 


3546 


0 


4380 


53 


48*25 


• 910 


844 


3643 


0 


4487 


54 


48*87 


• 905 


849 


3696 


0 


4545 


55 


5 1 *00 


• 927 


873 


3870 


0 


4743 



Er|c 142 



Off 


Average 


Overall 


Overall 


Average 


Average 


S-D. 


S-D, 


K 


Output 


%Prec- 


%Knov^ii 


°/6Prec- 


7oKnown 


%Prec- 


7oKnown 






ision 


Recall 


ision 


Recall 


ision 


Re ca 1 1 


0 


• 00 


*00 


*00 


*00 


• 00 


*00 


*00 


1 


• 40 


29*73 


*53 


11*83 


*58 


32*29 


1 *92 


2 


1 ^34 


35*20 


2*11 


23*66 


2*92 


37*17 


6*55 


3 


2^23 


35*75 


3*54 


27*78 


4*64 


35*33 


8*00 


4 


3^25 


32*45 


4*69 


30*40 


5*9 1 


34*54 


8*55 


5 


4^15 


27*20 


5*02 


28*31 


6 * 12 


33*20 


8*60 


6 


4^96 


27*98 


6*17 


28*37 


7*37 


31*01 


9*17 


7 


5^65 


27*62 


6*94 


29*36 


7*92 


3 1 *05 


9*37 


8 


6^69 


27*33 


8*13 


29*18 


9*83 


29*69 


10*9 1 


9 


7^62 


26*66 


9*04 


27*84 


10*50 


27*50 


11*32 


10 


8^74 


28*04 


10*9 1 


28*66 


13*15 


28*32 


15*75 


1 1 


9^39 


27*15 


11*34 


28*63 


13*63 


28*13 


15*80 


12 


10^58 


26*42 


12*44 


28*52 


14*95 


27*7 1 


16*13 


13 


1 1 • 18 


25*58 


12*73 


27*47 


15*21 


26*53 


16*16 


1 4 


12^05 


25*07 


13*44 


27*28 


16*66 


26*51 


18*29 


1 5 


13^34 


23*85 


14*16 


26*83 


17*48 


26*18 


18*35 


16 


14^44 


22*93 


14*74 


25*94 


17*84 


25*72 


18*29 


17 


14^90 


22*66 


15*02 


25*76 


18*17 


25*51 


18*53 


18 


1 5^75 


2 2*91 


15*36 


25*06 


18*79 


25*16 


18*93 


1 9 


16^98 


2 1 .03 


15*89 


24*0 1 


19*13 


24*51 


18*92 


20 


18^37 


2 1*90 


17*89 


23*46 


20*33 


23*96 


19*53 


21 


1 9 ^ 1 5 


2 1*50 


18*33 


23 * 1 3 


21*17 


23*25 


20 * 13 


22 


19^67 


2 1*54 


18*85 


23 * 14 


2 1*70 


23*07 


19*97 


23 


21 ^68 


21*23 


20*48 


22*65 


23*26 


22*86 


20*60 


24 


22^42 


2 1*58 


21*53 


22*4 1 


24*08 


22*67 


21*41 


25 


23^62 


20*80 


21*87 


21*69 


24*38 


2 1 *68 


21*38 


26 


23^95 


20*66 


22*01 


2 1*63 


24*5 1 


2 1 *68 


21*32 


27 


25^18 


20.79 


23*30 


2 1*93 


26*01 


2 1*37 


21*24 


28 


26^48 


20*46 


24 * 1 1 


21*72 


26*67 


21 *24 


21*20 


29 


27^24 


20*25 


24*55 


2 1*58 


27 * 13 


21 *20 


21*46 


30 


27^92 


20*29 


25*22 


21*64 


27*46 


2 1*27 


21*30 


3 1 


28*44 


. 19*92 


25*22 


21*61 


27*46 


2 1*28 


21*30 


32 


28*70 


19*90 


25*4 1 


2 1*54 


27*67 


2 1*22 


2 1*48 


33 


29*69 


19*78 


26 * 12 


2 1*36 


28*33 


21*21 


2 1*74 


34 


29*83 


1 9*75 


26*22 


2 1*37 


28*49 


2 1 *20 


2 1*76 


35 


30*03 


19*66 


26*27 


21*31 


28*54 


21*21 


2 1*75 


36 


3 0*63 


19*62 


26*75 


2 1 *23 


28*77 


2 1 *07 


21*66 


37 


32*28 


19*29 


27*70 


20*22 


29*74 


19*49 


21*81 


38 


33*90 


19*22 


29*00 


20*16 


30*85 


19*43 


22*12 


39 


35*31 


19*70 


30*96 


20*53 


31*44 


20*15 


22 * 12 


40 


36*73 


20*73 


33*88 


20*56 


32*52 


20*63 


22*62 


41 


37*49 


20*45 


34*11 


20*43 


32*77 


20*62 


22*57 


42 


38*30 


20*44 


34*83 


20*44 


33*3 2 


20*57 


22*50 


43 


39*74 


20*37 


36*03 


20*57 


34*11 


20*5 1 


22 * 10 


44 


40*95 


19*83 


36*12 


20*35 


34*27 


20*50 


22 * 10 


45 


41*47 


19*70 


36*36 


20*32 


34*60 


20*49 


22*40 


46 


43*03 


19*34 


37*03 


19*97 


35*07 


20*4 1 


22*41 


47 


43*61 


19*50 


37*85 


19*80 


35*61 


20*07 


22*89 


48 


44*52 


19*30 


38*23 


19*89 


36*70 


20*00 


22*89 


49 


45*15 


19*34 


38*85 


19*91 


37*21 


19*98 


22*90 


50 


46*30 


19*25 


39*67 


19*72 


37*76 


19*91 


22*95 


51 


46*89 


19*10 


39*86 


19*74 


38*41 


19*88 


22*81 


52 


47 * 10 


19*04 


39*90 


19*68 


38*42 


19*87 


22*81 


53 


48*25 


18*81 


40*38 


19*43 


38*8 1 


19*83 


22*72 


54 


48*87 


18*68 


40*62 


19*38 


39*07 


19*76 


22*66 


55 


51*00 


18*41 


4 1 *77 


16*63 


40*40 


18*09 


22*94 
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55 UNDERLINED REQUESTS, RUN 25(U13). TABLE RPK(55^25) 



Cut-off Average 
K Output 



0 


.00 


1 


.3 1 


2 


.98 


3 


1.98 


4 


2.55 


5 


3.04 


6 


4.16 


7 


4.84 


8 


5.18 


9 


5.82 


10 


6*85 


1 1 


7.51 


12 


8*36 


13 


8»78 


14 


9.47 


15 


9.47 


16 


10.33 


17 


10*33 


18 


11.11 


19 


1 1 .78 


20 


12.78 


21 


14.04 


22 


14*58 


23 


15.62 


24 


15.62 


25 


15.69 


26 


16*02 


27 


17.45 


28 


18*49 


29 


18.49 


30 


18*49 


3 1 


18.49 


32 


19*24 


33 


19.73 


34 


21.24 


35 


23.11 


36 


24*25 


37 


25.04 


38 


26*27 


39 


26*27 


40 


27.53 


41 


28.55 


42 


29.49 


43 


29*49 


44 


29.49 



O 




Rele- 

vant 


Irre 1- 
evant 


Not 

Assessed 


Output 


0 


0 


0 


0, 


1 0 


7 


0 


17* 


37 


17 


0 


* 54 


59 


50 


0 


1 09 


75 


65 


0 


140 


83 


84 


0 


167 


126 


103 


0 


229 


144 


122 


0 


266 


156 


1 29 


0 


265 


167 


153 


0 


320 


194 


163 


0 


377 


203 


210 


0 


413 


226 


234 


0 


460 


230 


253 


0 


483 


238 


283 


0 


521 


236 


283 


0 


521 


240 


328 


0 


56 8 


240 


328 


0 


568 


265 


.46 


0 


6 1 1 


287 


;6 1 


0 


648 


311 


392 


0 


703 


3 1 7 


455 


0 


772 


327 


475 


0 


802 


336 


523 


0 


859 


336 


523 


0 


859 


337 


526 


0 


863 


340 


54 1 


0 


88 1 


355 


605 


0 


960 


356 


660 


1 


1017 


356 


660 


1 


10 17 


356 


660 


1 


1017 


356 


660 


1 


1017 


378 


679 


1 


1058 


382 


697 


6 


1 085 


399 


763 


6 


1 168 


44 1 


824 


6 


1 27 1 


445 


883 


6 


1334 


445 


924 


8 


1 377 


453 


984 


8 


1445 


453 


984 


8 


1445 


462 


1044 


8 


1514 


470 


1 092 


8 


1570 


478 


1 130 


14 


1622 


478 


1 130 


14 


1622 


478 


1 130 


14 


1622 



kVk 

.000 

• 310 

.490 

• 660 

• 638 

• 608 

.693 

.691 

• 648 

.647 

• 685 

• 683 

.697 

.675 

.676 

• 63 1 

• 646 

• 608 

• 6 1 7 

• 620 

• 639 

• 669 

• 663 

• 679 

• 65 1 

• 628 

• 616 

• 646 

• 660 

• 638 

• 6 1 6 

• 596 

• 601 

• 598 

• 625 

• 660 

• 674 

• 677 

• 69 1 

• 674 

• 688 

.696 

• 702 

• 686 

• 670 



.144 



K 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

1 0 

1 1 

1 2 

1 3 

1 4 

1 5 

1 6 

1 7 

1 8 

1 9 

20 

2 1 

22 

23 

24 

25 

26 

27 

28 

29 

30 

3 1 

32 

33 

34 

35 

36 

37 

38 

39 

40 

4 1 

42 

43 

44 



Ave rage 


Overall 


Cve rail 


Average 


Average 


S.D. 




Output 


%Prec- 


%?Cnown 


% Prec - 


%Known 


7„P ree- 


4 Known 


K ' 


ision 


Recall 


ision 


Recall 


ls! on 


Recall 


.00 


.00 


.00 


.00 


• 00 


.00 


.00 


• 3 1 


58.82 


.7 1 


18.18 


1.92 


38.57 


7.10 


• 98 


68.52 


2.64 


34.24 


4.25 


44.09 


8.58 


1 .98 


54.13 


4.2 1 


39.76 


6.97 


4 1.50 


10.44 


2.55 


53.57 


5.35 


44.27 


7.98 


40.75 


10.38 


3.04 


49.70 


5.92 


43.97 


8.77 


39.25 


11.14 


4.16 


55.02 


8.99 


48.77 


1 0.96 


38.4 1 


11.74 


4.84 


54.14 


10.28 


5 1.50 


12.11 


37.54 


12.17 


5.18 


54.74 


11.13 


50.73 


13.42 


37.02 


13.98 


5.82 


52.19 


11.92 


50.22 


14.48 


36.12 


13.93 


6.85 


5 1.46 


13.85 


50.07 


16.67 


35.98 


16.15 


7.51 


49.15 


14.49 


49.07 


17.46 


35.95 


16.13 


8.36 


49.13 


16.13 


48.3 1 


19.13 


35.46 


17.03 


8.78 


47 0 62 


16.42 


48.6 1 


1 9 4 0 


3 5.11 


16.84 


9.47 


45.68 


16.99 


47.66 


20.04 


34.41 


17.24 


9.47 


45.68 


16.99 


47.66 


20.04 


34.4 1 


17.24 


10.33 


42.25 


17.13 


46.10 


20.52 


34.83 


17.56 


10.33 


42.25 


17.13 


46.10 


20.52 


34.83 


1 7.56 


11.11 


43.37 


18.92 


47.18 


22.04 


34.36 


19.22 


11.78 


44.29 


20.49 


47.06 


22.87 


34.29 


19.29 


12.78 


44.24 


22.20 


45.41 


23.85 


32.95 


19.40 


14.04 


4 1.06 


2 2.63 


44.03 


24.80 


33.53 


19.84 


14.58 


40.77 


23.34 


43.53 


25.45 


33.39 


20.07 


15.62 


39.12 


23.98 


42.82 


26.25 


33.50 


20.26 


15.62 


39.12 


23.98 


42.82 


26.25 


33.50 


20.26 


15.69 


39.05 


24.05 


42.78 


26.42 


33.50 


20.75 


16.02 


38.59 


24.27 


42.64 


26.69 


33.57 


20.82 


17.45 


36.98 


25.34 


42.79 


28.85 


33.31 


22.44 


18.49 


35.04 


25.4 1 


42.57 


28.98 


33.46 


22.46 


18.49 


35.04 


25.41 


42.57 


28.98 


33.46 


22.46 


18.49 


35.04 


25.41 


42.57 


28.98 


33.46 


22.46 


18.49 


■ 35.04 


25.4 1 


42.57 


28.98 


33.46 


22.46 


19.24 


35.76 


26.98 


42.43 


29.84 


33.39 


22.83 


19.73 


3 5.40 


27 . 27 


4 1.89 


30.14 


33.12 


23.26 


2 1.24 


34.34 


28.48 


40.86 


3 1.62 


32.14 


23.9 1 


23 . 1 1 


3 4.86 


3 1.48 


40.55 


33.68 


31.11 


24.00 


24.25 


33.5 1 


3 1.76 


40.24 


34.59 


3 1.36 


2 4.13 


25.04 


32.5 1 


3 1.76 


39.79 


34.59 


3 1.62 


24.13 


26.27 


3 1.52 


32.33 


38.26 


35.80 


30.70 


25.27 


26.27 


3 1.52 


32.33 


38.26 


35.80 


30.70 


25.27 


27.53 


30.68 


32.98 


38.14 


36.76 


30.78 


25 . 1 2 


28.55 


30.09 


33.55 


37.10 


37.28 


30.20 


25.38 


29.49 


29.73 


34.12 


36.79 


37.72 


30.25 


25.26 


29.49 


29.73 


34.12 


36.79 


37.72 


30.25 


25.26 


29.49 


29.73 


34.12 


36.79 


37.72 


30.25 


25.26 
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off 

K 


Average 

□utput 

K' 


k'/k 


45 


29.49 


.655 


46 


29.49 


.641 


47 


29.49 


.627 


48 


29.49 


.6 1 4 


49 


29.93 


.6 11 


50 


30.47 


.609 


5 1 


30.47 


.597 


52 


30.47 


.586 


53 


30.47 


.575 


54 


3 1.78 


.589 


55 


3 1.78 


.578 


56 


3 1.78 


.568 


57 


3 1.78 


.558 


58 


33.69 


.58 1 


59 


3 3.69 


.57 1 


60 


33.69 


.562 


6 1 


35 . 15 


.576 


62 


37.49 


.605 


63 


38.78 


.616 


64 


38.78 


.606 


65 


38.78 


.597 


66 


40.67 


.6 16 


67 


40.67 


.607 


68 


42.78 


.629 


69 


42.78 


.620 


70 


43.89 


.627 


7 1 


44.27 


.624 


72 


44.27 


.615 


73 


4 6.25 


.634 


74 


46.25 


.625 


75 


48.29 


.644 


76 


48.29 


.635 


77 


48.29 


.627 


78 


48.29 


.6 19 


79 


48.82 


.618 


80 


48.82 


.610 


6 1 


50.40 


.622 


82 


50.40 


.615 



Irrel- 

evant 


Not 

Assessed 


Output 


1 130 


14 


1622 


1 130 


14 


1622 


1 130 


14 


1622 


1 130 


14 


1622 


1 147 


14 


1646 


1 169 


1 8 


1676 


1169 


18 


1676 


1169 


1 8 


1676 


1 169 


1 8 


1676 


1203 


49 


1748 


1203 


49 


1748 


1203 


49 


1748 


1203 


49 


1748 


1250 


92 


1853 


1250 


92 


1853 


1250 


92 


1853 


1268 


146 


1933 


1327 


193 


2062 


1361 


230 


2 133 


1 36 1 


230 


2 133 


1 36 1 


230 


2133 


14 13 


280 


2237 


1413 


280 


2237 


142 1 


387 


2353 


142 1 


387 


2353 


1456 


409 


24 1 4 


1458 


428 


2435 


1458 


428 


2435 


1494 


486 


2544 


1494 


486 


2544 


1 557 


52 1 


2656 


1 557 


52 1 


2656 


1 557 


52 1 


2656 


1557 


52 1 


2656 


1 558 


548 


2685 


1 558 


548 


2685 


1589 


604 


2772 


1589 


604 


2772 



Rele- 

vant 

478 

478 

478 

478 

485 

489 

489 

489 

489 

496 

496 

496 

496 

5 I I 

51 I 

5 I I 

5 I 9 

542 

542 

542 

542 

544 

544 

545 

545 

549 

549 

5 19 

564 

564 

578 

578 

578 

578 

579 

579 

579 

579 

a46 



ff 

K 

45 

46 

47 

48 

49 

50 

5 I 

52 

53 

54 

55 

56 

57 

58 

59 

60 

6 1 

62 

63 

64 

65 

66 

67 

68 

69 

70 

7 I 

72 

73 

74 

75 

76 

77 

78 

79 

80 

0 I 

02 



Ave rage 
Output 


Ove rail 
%Prec- 
Islon 


Ove rail 
%Known 
Recall 


Ave rage 
VoPrec- 
Islon 


Average 
7o Knovm 
Recall 


S. D. 

%Prec- 

Islon 


S. D- 
%Known 
Recall 


29.49 


29.73 


34.12 


36.79 


37.72 


30.25 


25.26 


29.49 


29.73 


34.12 


36.79 


37.72 


30.25 


25.26 


29.49 


29.73 


34.12 


36.79 


37.72 


30.25 


25.26 


29.49 


29.73 


34.12 


36.79 


37.72 


30.25 


25.26 


29.93 


29.72 


34.62 


36.72 


38.18 


30.25 


25.69 


30.47 


29.49 


34.90 


36.66 


38.54 


30.29 


25.86 


30.47 


29.49 


34.90 


36.66 


38.54 


30.29 


25.86 


30.47 


29.49 


34.90 


36.66 


38.54 


30.29 


25.86 


30.47 


29.49 


34.90 


36.66 


38.54 


30.29 


25.86 


3 1.78 


29.19 


35.40 


36.42 


38.93 


30.34 


25.71 


3 1.78 


29.19 


35.40 


36.42 


38.93 


30.34 


25.71 


3 1.78 


29.19 


35.40 


36.42 


38.93 


30.34 


25.7 1 


3 1 7 8 


29.19 


35.40 


36.42 


38.93 


30.34 


25.7 1 


33.69 


29.02 


36.47 


36.0 1 


39.45 


30.27 


25.62 


33.69 


29.02 


36.47 


36.0 1 


39.45 


30.27 


25.62 


33.69 


29.02 


36.47 


36.0 1 


39.45 


30.27 


25.62 


35.15 


29.04 


37.04 


35.92 


40.22 


30.26 


26.3 1 


37.49 


29.00 


38.69 


35.05 


41.11 


29.63 


26.82 


38.78 


28.48 


30.69 


34.98 


41.11 


29.7 1 


26.82 


38.78 


28.48 


30.69 


34.98 


41.11 


29.7 1 


26.82 


38.78 


28.48 


38.69 


34.98 


4 1.11 


29.7 1 


26.82 


40.67 


27.80 


30.83 


34.24 


4 1.37 


29.73 


26.97 


40.67 


27.80 


30.83 


34.24 


4 1.37 


29.73 


26.97 


42.78 


27.72 


38.90 


33.64 


4 1.39 


29.17 


26.95 


42.78 


27.72 


38.90 


33.64 


4 1.39 


29.17 


26.95 


43.89 


27.38 


39 . 19 


33.6 1 


4 1.91 


29.19 


27.11 


44.27 


27.35 


39.19 


33.57 


4 1.91 


29.14 


27.11 


44.27 


27.35 


39.19 


33.57 


41.91 


29.14 


27.11 


46.25 


. 27.4 1 


40.26 


33.37 


42.59 


29.10 


27.02 


46.25 


27.4 1 


40.26 


33.37 


42.59 


29.10 


27.02 


48.29 


27.07 


41.26 


32.24 


43.39 


27.89 


27.95 


48.29 


27.07 


4 1.26 


32.24 


43.39 


27.89 


27.95 


48.29 


27.07 


4 1.26 


32.24 


4 3-39 


27.89 


27.95 


48.29 


27.07 


41.26 


32.24 


43.39 


27.89 


27.95 


48.82 


27.09 


4 1.33 


32.26 


43.48 


27.88 


28.01 


48.82 


27.09 


4 1.33 


32.26 


4 3.48 


27.88 


28.0 1 


50.40 


26.7 1 


4 1.33 


32.13 


43.48 


27.98 


28.0 1 


50.40 


26.7 1 


4 1.33 


3 2.13 


43.48 


27.98 


28.0 1 
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APPENDIX B8 CONTIlvfUED 



In this and the following two tables, documents not assessed 
in any standard or manual strategy have been set Irrelevant. 



93 REQUESTS, RUN 13(KWS) CONTINUED. TABLE RPL(93, 13)- 



-Off 


A ve rage 


k'/k 


Rele- 


Irrel- 


Output 


Ove rail 


Ove rail 


K 


Output 




vant 


evant 




%Prec- 


7oKnovm 




K' 










ision 


Recall 


73 


51.66 


.708 


1042 


3762 


4804 


2 1.69 


49.86 


74 


5 1.66 


.698 


1042 


3762 


4804 


2 1.69 


49.86 


75 


5 1.66 


.689 


1042 


3762 


4804 


2 1.69 


4 9 . -8 6 


76 


52.56 


.692 


1047 


384 1 


4888 


21 .42 


50.10 


77 


52.56 


.683 


1047 


384 1 


4888 


21 .42 


50.10 


78 


52.56 


.674 


1047 


384 1 


4888 


21.42 


50.10 


79 


52.56 


.665 


1047 


384 1 


4888 


2 1.42 


50.10 


80 


52.56 


.657 


1047 


384 1 


4888 


21 .42 


50.10 


a 1 


52.56 


.649 


1047 


384 1 


4888 


21.42 


50*10 


82 


5 4.02 


.659 


1059 


3965 


5024 


2 1.08 


50*67 


83 


54.02 


.65 1 


1059 


3965 


5024 


21 .08 


50*67 


84 


54.02 


.643 


1059 


3965 


5024 


2 1.08 


50*67 


85 


55.75 


.656 


1062 


4 1 23 


5 1 85 


20.48 


50*81 


66 


55.7 5 


.648 


1062 


4123 


5 1 85 


20.48 


50*8 1 


87 


57o4 1 


.660 


1064 


4275 


5339 


19.93 


50*91 


88 


57.4 1 


.652 


1064 


427 5 


5339 


19.93 


50*9 1 


89 


57.41 


.645 


1064 


4275 


5339 


19.93 


50*9 1 


90 


58.89 


.654 


1085 


4392 


5477 


19.81 


5 1.91 


9 1 


58.89 


.647 


1085 


4392 


5477 


19.81 


51.9 1 


92 


60.47 


o657 


1 1 00 


4524 


5624 


1 9.56 


52.63 


93 


60 . 47 


.650 


1 100 


4524 


5624 


19.56 


52.63 


94 


62.06 


.660 


1110 


4662 


5772 


19.23 


53. 1 1 


95 


62.06 


.653 


1110 


4662 


5772 


19.23 


53. 1 1 


96 


62.06 


.646 


1110 


4662 


5772 


19.23 


53.11 


97 


62.06 


.640 


1110 


4662 


5772 


19.23 


53.11 


98 


65.04 


.664 


1 1 44 


4905 


6049 


18.91 


54.7^ 


99 


65.04 


.657 


1144 


4905 


6049 


18.91 


54.74 


100 


65.04 


.650 


1 144 


4905 


6049 


18.91 


54.74 


101 


65.04 


.644 


1 1 44 


4905 


6049 


18.91 


54.74 


102 


65.04 


.638 


1 1 44 


4905 


6049 


18.91 


54.74 


103 


65.04 


.631 


1 1 44 


4905 


6049 


18.91 


54.74 


104 


67.10 


.645 


1158 


5082 


6240 


18.56 


55.4 1 


105 


67.10 


€ 639 


1 158 


5082 


6240 


18*56 


55.4 1 


106 


67.10 


.633 


1 158 


5082 


6240 


18.56 


55.4 1 


1 07 


67.10 


.627 


1158 


5082 


6240 


18.56 


55.4 1 


108 


67.10 


.62 1 


1 158 


5082 


6240 


18*56 


55.4 1 


109 


67.10 


.616 


1 158 


5082 


6240 


18*56 


55.41 


1 10 


69.27 


• 630 


1 1 66 


5276 


6442 


18*10 


55.79 


1 1 1 


71.49 


• 6 4 4 


1 1 67 


5482 


6649 


17*55 


55.84 


1 12 


73.83 


.659 


1184 


5682 


6866 


17.24 


56.65 


1 1 3 


75.97 


• 672 


1 185 


5880 


7065 


16.77 


56.70 


1 1 4 


75.97 


• 666 


1 1 85 


5880 


7065 


16.77 


56.70 


1 15 


75.97 


• 66 1 


1 1 85 


5880 


7065 


16*77 


56.70 


1 16 


79.45 


• 685 


1204 


6185 


7389 


16*29 


57.6 1 


1 17 


79.45 


• 679 


1204 


6185 


7389 


16.2'^ 


57.6 1 


118 


81.43 


• 690 


1221 


6352 


7573 


16.12 


58.42 
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Cut-Off 


Average 


kVk 


Rele- 


Irrel- 


Output 


□ ve rail 


Overall 


K 


Output 




vant 


evant 




\?re el- 


%Knovm 




K' 










ision 


Recall 


1 1 9 


6U43 


.664 


1221 


6352 


7573 


16.12 


56.42 


120 


61 .43 


.679 


1221 


6352 


7573 


16.12 


56.42 


1 2 1 


61 .43 


.673 


1221 


6352 


7573 


16.12 


50.42 


122 


03.70 


.607 


1225 


6567 


7792 


15.72 


56.6 1 


123 


03.70 


.681 


1225 


6567 


7792 


15.72 




124 


03.70 


.676 


1225 


6567 


7792 


15.72 


56.61 


125 


66.00 


.669 


1226 


6779 


0005 


15.32 


56.66 


126 


66.00 


.633 


1226 


6779 


0005 


15.32 


56.66 


127 


66.00 


.670 


1226 


6779 


0005 


15.32 


50.66 


120 


06.08 


.673 


1226 


6779 


0005 


15.32 


50.66 


129 


90.76 


.704 


1259 


7102 


644 1 


14.9? 


6 0.24 


1 30 


92.95 


o7l5 


1266 


7376 


0644 


14.67 


60.67 


131 


95:^37 


• 72 0 


1277 


7592 


0069 


14.40 


61.10 


132 


95.37 


.723 


1277 


7592 


0069 


14.40 


61.10 


133 


97. 1 9 


• 731 


1290 


7749 


9039 


14.27 


61 .72 


134 


99.14 


.740 


13 13 


7907 


0920 


14.24 


6 2.62 


135 


99.14 


.734 


13 13 


7907 


9220 


14.24 


6 2.62 


136 


99.14 


.729 


1313 


7907 


9220 


14.24 


62.02 


137 


99.14 


.724 


13 13 


7907 


9220 


14.24 


6 2.62 


1 38 


99. 1 4 


.710 


1313 


7907 


9220 


14.24 


62.82 


1 39 


103.75 


.746 


1352 


0297 


9649 


14.01 


64.69 


140 


106.34 


.760 


1356 


0534 


9090 


13.71 


64.66 


1 4 1 


106.34 


.754 


1356 


0534 


9090 


13.71 


64.66 


142 


106.34 


.749 


1356 


0534 


9090 


13.7 1 


64.66 


143 


107.00 


.754 


1362 


0663 


10025 


13.59 


65.17 


144 


1 07.00 


.749 


1362 


0663 


1 0025 


13.59 


65.17 


145 


107.00 


.743 


1362 


0663 


1 0025 


13.59 


65.17 


146 


107.00 


.738 


1362 


0663 


1 0025 


13.59 


65.17 


147 


107.00 


.733 


1362 


0663 


1 0025 


13.59 


65.17 


140 


107.00 


.720 


1362 


6663 


1 0025 


13.59 


65.17 


149 


107.00 


.723 


1362 


0663 


1 0025 


13.59 


65. 17 


150 


107.00 


.719 


1362 


8663 


1 0025 


13.59 


65.17 


151 


110.63 


.733 


1 367 


0922 


10209 


13.29 


65.4 1 


152 


112.14 


.730 


1375 


9054 


10429 


13.16 


65.79 


153 


112.14 


.733 


1375 


9054 


10429 


13.16 


65.79 


154 


115.09 


.753 


1401 


9377 


1 0770 


13.00 


67.03 


155 


115.09 


.740 


1401 


9377 


1 0770 


13.00 


67.03 


156 


115.09 


.743 


1401 


9377 


1 0770 


13.00 


67.03 


157 


115.09 


.730 


1401 


9377 


1 0778 


13.00 


67.03 


150 


115.09 


.733 


1401 


9377 


1 0770 


13.00 


6 7.03 


159 


115.09 


.729 


1401 


9377 


1 0770 


13.00 


6 7.03 


1 60 


121.02 


.756 


14 15 


9040 


11255 


12.57 


6 7.70 


161 


123.97 


.770 


1430 


1 0099 


1 1529 


12.40 


68.42 


162 


1 23.97 


.765 


1430 


1 0099 


1 1 529 


12.40 


68 42 


163 


126.73 


.777 


1473 


10313 


11706 


12.50 


70.40 


164 


126.73 


.773 


1473 


10313 


1 1706 


12.50 


70.40 


165 


126.73 


.760 


1473 


10313 


1 1706 


12.50 


70.40 


166 


1 26.73 


.763 


1473 


10313 


11706 


12.50 


70.40 


167 


1 26.73 


.759 


1473 


10313 


1 1706 


12.50 


70.40 


160 


126.62 


.767 


1474 


1 0506 


1 1 900 


12.30 


70.53 


1 69 


1 26.82 


.762 


1474 


1 0506 


1 1900 


12.30 


70.53 


170 


126.62 


.750 


1474 


1 0506 


1 1900 


12.30 


70.53 


171 


131.01 


.766 


1 478 


1 0706 


12 184 


12.13 


70.72 


172 


131.01 


.762 


1470 


1 0706 


12164 


12.13 


70.72 


173 


131.01 


• 757 


1478 


1 0706 


12104 


12.13 


70.72 


174 


133.34 


.766 


1405 


10916 


124r 1 


11.97 


7 1.05 


175 


1 33.34 


• 762 


1405 


10916 


12401 


11.97 


7 1.05 


176 


136.39 


• 775 


1466 


11196 


1 2664 


11.73 


71.20 


177 


136.39 


.77 1 


1466 


11196 


1 2604 


1 1 .73 


7 1.20 


176 


136.39 


• 766 


1 400 


11196 


12604 


11.73 


71.20 


a 


136.39 


.762 


1466 


11196 


12604 


11.73 


71.20 


c R I 16 0 


1 36.39 


• 750 


146 6 


11196 


1 2604 


1 1 o73 


71.20 


ciyL 18 1 


136.39 


• 754 


1 408, 


11196 


12664 


11.73 


71.20 




136.39 


• 749 


1480 


1 1 196 


12664 


11.73 


71.20 
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0 

1 


Average 

Output 

139*63 


kVk 


183 


.763 


) 84 


139.63 


.759 


185 


139.63 


.755 


186 


139.63 


.75 1 


187 


139.63 


.747 


188 


139.63 


.743 


189 


142.56 


.754 


190 


146.23 


.770 


19 1 


146.23 


.766 


192 


149.40 


.778 


193 


149.40 


.774 


194 


153.30 


.790 


195 


156.46 


.802 


196 


156.46 


.798 


197 


156.46 


.794 


198 


160.19 


.809 


199 


163.81 


.823 


200 


M63.8 1 


.8 19 


210 


169.53 


.807 


220 


183.20 


.833 


230 


190.43 


.828 


240 


199.99 


.833 


250 


206.68 


.827 


260 


217.72 


.837 


270 


224.67 


.832 


280 


229.53 


.820 


290 


229.53 


.79 1 


300 


234.43 


.78 1 


3 1 0 


250.28 


.807 


320 


260.65 


.815 


330 


266.01 


.806 


340 


270.99 


.797 


350 


270.99 


.774 


360 


276.41 


.768 


370 


281.54 


.76 1 


380 


287.67 


.757 


390 


287.67 


.738 


400 


295.04 


.738 


4 10 


295.04 


.720 


420 


306.63 


.730 


430 


306.63 


.7 13 


440 


306.63 


.697 


450 


331.73 


.737 


460 


345.68 


.75 1 


470 


365.96 


.779 


480 


365.96 


.762 


490 


365.96 


.747 


500 


365.96 


.732 



o 




I 



Irrel- 


Output 


□ve rail 


□ve rail 


evant 




7oPrec- 


%Know:ri). 






Islon 


Recall 


1 1495 


12986 


11.48 


7 1 .34| 


1 1495 


12 986 


11.48 


7 1.34! 


1 1495 


1 2986 


11.48 


7 1 . 3 4| 


1 1495 


12986 


1 1 .48 


7 1.3 4\ 


1 1495 


12986 


11.48 


7 1 .34\ 


1 1495 


12986 


11.48 


71.34 \ 


1 1748 


13258 


11.39 


72.25 ' 


12082 


13599 


11.16 


72.58 


12082 


13599 


11.16 


72.58 ! 


1 2363 


1 3694 


1 1 .02 


73.25 ’ 


12363 


13894 


1 1 .02 


73.25 ; 


1 2723 


14257 


10.76 


73.40 1 


130 14 


1455 1 


10.56 


73.54 : 


13014 


1455 1 


10.56 


73.54 1 


13014 


14551 


10.56 


73.54 i 


13355 


14898 


10.36 


73.83 1 


13 6/9 


15234 


10.2 1 


74.40 1 


13679 


15234 


10.2 1 


7 4.40 j 


1420 1 


1 5766 


9.93 


74.88 1 


15446 


17038 


9.34 


76. 17 j 


16 104 


17710 


9.07 


76.84 1 


16967 


18599 


8.77 


78.09 I 


17584 


19221 


8.52 


78.33 ! 


18585 


20248 


6.2 1 


79.57 


192 19 


20894 


8.02 


80.14 1 


19670 


2 1346 


7.85 


80.19 j 


19670 


2 1346 


7.85 


80.19 1 


20 123 


2 1 802 


7.70 


80.33 I 


2 1589 


23276 


7.25 


80.72 j 


22550 


24240 


6.97 


80.86 i 


23043 


24739 


6.86 


81.15 I 


?3498 


25202 


6.76 


81.53 i 


23498 


25202 


6.76 


81.53 1 


23993 


25706 


6.66 


81.96 ; 


244 1 5 


26 183 


6.75 


84.59 i 


24966 


26753 


6.68 


85.50 j 


24966 


26753 


6.68 


85.50 j 


25648 


27439 


6.53 


85.69 j 


25648 


27439 


6.53 


85.69 


2672 1 


28517 


6.30 


85.93 


26721 


285 17 


6.30 


85.93 


2672 1 


28517 


6.30 


85.93 


29017 


30851 


5.94 


87.75 


30302 


32148 


5.74 


88.33 


32 172 


34034 


5.47 


89.09 


32 172 


34034 


5.47 


89.09 


32 172 


34034 


5.47 


89.09 


32 172 


34034 


5.47 


89.09 



Rele- 

vant 

149 1 

1 49 1 

1 49 1 

149 1 

149 1 

149 1 

1510 

1517 

1517 

1 53 1 

1 53 1 

1534 

1 537 

1 537 

1 537 

1 543 

1555 

1555 

1 565 

1 592 

1606 

1632 

1637 

1663 

1675 

1676 

1676 

1679 

1687 

1690 

1696 

1704 

1704 

17 13 

1768 

1787 

1 787 

1791 

1791 

1796 

1796 

1796 

1834 

1846 

1862 

1862 

1862 

1862 
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93 REQUESTS, RUN l^(MCSOl) CONTINUED, TABLE RPL(93^ 





Cut-off 

K 


Average 

Output 

K' 


K'/K 




55 


51*00 


*927 




56 


52*74 


*942 




57 


53*76 


*943 




50 


54*25 


*935 




59 


55*30 


*937 




60 


55*54 


*926 




61 


56* 1 1 


*920 




62 


57*41 


*926 




63 


57*43 


*912 




64 


57*43 


*097 




65 


50*06 


*906 




66 


59*15 


* 096 




67 


59*10 


*003 




60 


59*90 


*00 1 




69 


60*43 


*076 




70 


6 1*02 


*003 




7 1 


62*30 


*077 




72 


64*25 


*092 




73 


65*32 


*095 




74 


65*96 


* 09 1 




75 


66* 1 9 


*003 




76 


66*9 1 


*000 




77 


60*9 1 


*095 




70 


69*60 


*092 




79 


7 1*39 


*904 




00 


73*75 


*922 




0 1 


75*14 


*920 




0 2 


76*60 


*935 




03 


77*13 


*929 




04 


79*61 


*940 




05 


0 0*37 


*946 




06 


0 1*40 


*947 




07 


02*97 


*954 




00 


04*73 


*963 




09 


05*00 


*955 




90 


05*06 


*954 




9 1 


06*57 


• 951 




92 


06*04 


*944 




93 


60*10 


*947 




94 


09*90 


*956 




95 


90*39 


*951 




96 


91*43 


*952 




97 


93* 10 


*961 




90 


94*66 


*966 




99 


95*22 


*962 




100 


95*05 


*959 




101 


96*9 1 


*960 




102 


97*65 


• 957 




1 03 


90*05 


*952 




104 


98*95 


*951 




105 


100*20 


*954 




1 06 


100*45 


*940 




107 


102*60 


*959 




100 


102*97 


*953 




109 


103*07 


*953 




1 1 0 


104*12 


*947 




1 1 1 


104*43 


*941 




1 1 2 


104*72 


*935 




1 13 


106* 14 


*939 




1 14 


106*20 


*932 




1 15 


106*40 


*925 




1 16 


106*40 


*9 17 


o 


1 17 


106*95 


*914 


ERIC 


1 10 


107*53 


.91 1 



Irrel- 


exit put 


Overall 


Overall 


evant 




Prec- 


% Known 






ision 


Recall 


3070 


4743 


10.41 


4 1*77 


4029 


4905 


17.06 


41*91 


4 1 07 


5000 


17.06 


42*73 


4 150 


5045 


17.74 


42*02 


4232 


5143 


17.71 


43*59 


4253 


5 1 65 


17.66 


43*64 


4290 


5210 


17.63 


44*02 


4400 


5339 


17.59 


44*93 


4402 


5341 


17.50 


44*93 


4402 


5341 


17.50 


44*93 


4530 


5474 


17.25 


45*17 


4556 


5501 


17.10 


45*22 


4550 


5504 


17.19 


45*26 


4620 


5571 


17.07 


45*50 


4667 


5620 


16.96 


45*60 


4783 


5749 


16.00 


46*22 


4020 


5794 


16*67 


46*22 


4997 


5975 


16.37 


46*79 


5006 


6075 


16.20 


47*32 


5143 


6134 


16.16 


47*42 


5161 


6156 


16.16 


47*61 


5223 


6223 


16.07 


47*05 


5399 


6409 


15.76 


40*33 


5462 


6473 


15.62 


40*37 


5603 


6639 


15.60 


49;57 


50 10 


6059 


15.10 


49*0 1 


5944 


6900 


14.94 


49*95 


6004 


7131 


14.60 


50* 10 


6120 


7173 


14.60 


50*30 


6345 


7404 


14.30 


50*67 


6413 


7474 


14.20 


50*77 


65 1 1 


7570 


14.00 


51*05 


6646 


77 16 


13.07 


5 1*20 


6001 


7000 


13.69 


5 1*63 


6025 


7905 


13.66 


5 1*67 


6905 


7905 


13.53 


51*67 


6962 


0051 


13.53 


52* 1 1 


6907 


0076 


13.40 


52*11 


7096 


0 193 


13.39 


52*49 


7262 


036 1 


13.14 


52*50 


7307 


0406 


13*07 


52*50 


7401 


0503 


12.96 


52*73 


756 1 


0666 


12.75 


52*07 


7692 


0003 


12.62 


53* 16 


7744 


0055 


12.55 


53* 16 


7795 


09 1 4 


12.55 


53*54 


7 076 


9013 


12.62 


54*40 


7940 


9001 


12.56 


54*59 


7970 


9 119 


12.51 


54*59 


0057 


9202 


12.44 


54*70 


0 173 


93 19 


12.30 


54*83 


0 195 


9342 


12.20 


54*00 


0309 


9542 


12.00 


55* 17 


8423 


9576 


o 

• 

C\J 


55*17 


0502 


9660 


11.99 


55*41 


0523 


9603 


11.90 


55*50 


055 1 


9712 


1 1.95 


55*55 


0570 


9739 


11.92 


55*55 


0709 


9071 


11.77 


55*60 


0722 


9004 


1 1.76 


55*60 


0732 


9095 


1 1.75 


55*65 


0732 


9095 


1 1.75 


55*65 


0783 


9946 


1 1.69 


55*65 


0035 


lOOOO 


11.65 


55*74 



Rele- 

vant 

073 

076 

093 

095 

9 1 1 

9 12 

920 

939 

939 

939 

944 

945 

946 

95 1 

953 

966 

966 

970 

909 

99 1 

995 

1000 

1010 

1011 

1036 

104 1 

1044 

1047 

1 053 

1 059 

1061 

1067 

1070 

1079 

1000 

1000 

1 009 

1009 

1 097 

1 099 

1099 

1 1 02 

1 1 05 

1 1 1 1 

1 1 1 1 

1119 

1 137 

1141 

1141 

1 1 45 

1 1 46 

1 147 

1 153 

1 153 

1 150 

1 160 

1161 

1161 

1 162 

1 162 

1 163 

1 163 

1163 

1 165 



ERIC it52 



-Off 


A ve rage 


k'/k 


Rele- 


Irrel- 


Output 


Overall 


Overall 


K 


Output 




vant 


evant 




% Prec- 


% Known 














ision 


'Recall 


1 19 


109*60 


*92 1 


1 170 


9023 


10 193 


11*48 


55*98 


1 20 


110*28 


*919 


1171 


9085 


1 0256 


11*42 


56*03 


1 2 1 


110*85 


*916 


1 173 


9136 


1 0309 


11*38 


56*12 


122 


111*18 


*91 1 


1 175 


9 1 65 


1 0340 


11*36 


56*22 


123 


113*00 


*919 


1 1 85 


9324 


1 0509 


11*28 


54 *70 


124 


113*11 


*912 


1 1 85 


9334 


10519 


11*27 


56*70 


1 25 


114*11 


*913 


1 1 87 


9425 


10612 


11*19 


56 * 79 - 


126 


1 15*39 


*916 


1 188 


9543 


1 073 1 


11*07 


56*84 


127 


1 15*39 


*909 


1 1 88 


9543 


1 073 1 


11*07 


56*84 


128 


1 15*40 


*902 


1 188 


9544 


1 0732 


11*07 


56*84 


129 


115*82 


*898 


1 190 


958 1 


1 077 1 


11*05 


56*94 


130 


115*82 


*891 


1 190 


958 1 


1 077 1 


11*05 


56*94 


131 


117*73 


*899 


1 194 


9755 


1 0949 


10*9 1 


57 * 13 


132 


1 19*97 


*909 


1206 


995 1 


11157 


10*81 


57*70 


1 33 


122*08 


*918 


121 ^ 


10 143 


1 1 353 


10*66 


57*89 


134 


122*92 


*917 


121 


1 0222 


1 1432 


10*58 


57*89 


135 


124*20 


*920 


12^5 


1 0335 


1155 1 


10*53 


58*18 


1 36 


125*77 


*925 


12 17 


10480 


1 1697 


10*40 


58*23 


1 37 


128*68 


*94 1 


1232 


1 0754 


1 1986 


10*28 


58*95 


138 


130*97 


*949 


1233 


1 0947 


12 180 


10*12 


59*00 


1 39 


131*15 


*944 


1233 


1 0964 


12 197 


10*11 


59*00 


140 


131*75 


*94 1 


1 255 


1 0998 


1 2253 


10*24 


60*05 


141 


134*97 


*957 


1 256 


1 1296 


1 2552 


10*01 


60*10 


142 


135*67 


*955 


1256 


11361 


12617 


9*95 


60*10 


143 


136*73 


*956 


1 258 


1 1 458 


12716 


9*89 


60*19 


1 44 


137*16 


*953 


1264 


1 1 492 


1 2756 


9*9 1 


60*48 


1 45 


138*11 


*952 


1 272 


1 1572 


1 2844 


9*90 


6 0*86 


146 


138*28 


*947 


1275 


1 1585 


1 2860 


9*9 1 


6 1*00 


147 


138*69 


*943 


1 275 


1 1623 


1 2898 


9*89 


6 1*00 


148 


139*43 


*942 


1 277 


1 1690 


12967 


9*65 


61*10 


149 


141*00 


*946 


1277 


1 1 836 


13113 


9*74 


61*10 


150 


142*87 


*952 


1286 


1 200 1 


13287 


9*68 


61*53 


151 


143*57 


*95 1 


1 287 


1 2065 


1 3352 


9*64 


61*58 


152 


143*57 


*945 


1287 


1 2065 


1 3352 


9*64 


61 *58 


153 


143*57 ■ 


*938 


1287 


1 2065 


1 3352 


9*64 


61 *58 


154 


144*05 


*935 


1287 


12110 


1 3397 


9*61 


61 *58 


155 


144*82 


*934 


1289 


12179 


13468 


9*57 


6 1*67 


156 


144*82 


*928 


1289 


12 179 


13468 


9*57 


6 1*67 


157 


146*39 


*932 


1 292 


1 2322 


13614 


9*49 


61*82 


158 


147*02 


*93 1 


1292 


1 2381 


1 3673 


9*45 


61 *62 


1 59 


147*02 


*925 


1292 


1 238 1 


1 3673 


9*45 


61*82 


160 


147*02 


*919 


1292 


1 2381 


1 3673 


9*45 


61 *62 


161 


149*98 


*932 


1296 


12652 


13948 


9*29 


62*01 


162 


152*01 


*938 


1299 


1 2838 


14 137 


9*19 


62 * 15 


163 


152*01 


• 933 


1299 


1 2838 


14 137 


9*19 


62*15 


164 


152*86 


*932 


1 299 


12917 


14216 


9*14 


62*15 


165 


153*75 


*932 


1 299 


1 3000 


14299 


9*08 


62 * 15 


166 


155*57 


*937 


1302 


13 166 


1 4468 


9*00 


62*30 


167 


155*70 


*932 


1302 


13178 


1 4480 


8*99 


62*30 


168 


156*88 


*934 


13 10 


13280 


1 4590 


8*98 


62*68 


169 


160*27 


*948 


1313 


1 3592 


1 4905 


8*8 1 


62*62 


170 


164*17 


*966 


1314 


1 3954 


15268 


8*61 


62*87 


171 


165*74 


*969 


1314 


14100 


15414 


8*52 


62*87 


172 


166*78 


*970 


1316 


14 195 


15511 


8*48 


62*97 


173 


166*78 


*964 


1316 


14 195 


15511 


8*48 


62*97 


174 


166*66 


*969 


1326 


1 4361 


13687 


8*45 


63*44 


1 75 


171*19 


*978 


1327 


1 4594 


15921 


8*33 


63*49 


176 


171 *60 


*975 


1329 


14630 


1 5959 


8*33 


63*59 


177 


175*23 


*990 


1331 


14965 


16296 


8*17 


6 3*68 


178 


176*03 


*989 


1333 


15038 


1637 1 


8*14 


63*78 


179 


176*03 


*983 


1333 


15038 


1 637 1 


8*14 


63*78 


180 


177*53 


*986 


1335 


15175 


16510 


8*09 


63*88 


1 8 1 


177*53 


*981 


1335 


15175 


16510 


8*09 


63*88 


182 


177*53 


*975 


1335 


15175 


16510 


8*09 


63*88 



i 

o 


Ave rage 
Output 


k'/k 


Rele* 

vant 


183 


177.53 


.970 


1 335 


184 


179.90 


.978 


1337 


185 


180.14 


.974 


1 337 


186 


180.91 


.973 


1341 


187 


183.38 


.98 1 


1355 


188 


185.04 


.984 


1356 


189 


185.47 


.98 1 


1357 


190 


185.73 


.978 


1357 


19 1 


187.47 


.982 


1358 


192 


189.01 


.964 


1365 


1 93 


189.74 


.983 


1367 


194 


189.74 


.978 


1 367 


195 


193.92 


.994 


1 368 


196 


194.00 


.990 


1368 


197 


196.15 


.996 


1 373 


198 


196.52 


.993 


1375 


199 


196.52 


.988 


1375 


200 


197.31 


.987 


1377 


210 


204.38 


.973 


1391 


220 


216.24 


.983 


1403 


230 


228.29 


.993 


1424 


240 


234.24 


.976 


1437 


250 


242.77 


.97 1 


1 450 


260 


255. 15 


.98 1 


1455 


270 


263.59 


.976 


1 468 


280 


275.69 


.985 


1480 


290 


284.68 


.982 


1489 


300 


291.96 


.973 


1494 


310 


300.75 


.970 


1501 


320 


305.84 


o 956 


1508 


330 


316.05 


.958 


1524 


340 


320.73 


.943 


1534 


350 


328 . 16 ' 


.936 


1544 


360 


340.55 


.946 


1549 


370 


345. 16 


.933 


1554 


380 


363.46 


.956 


1559 


390 


369.87 


.948 


1582 


400 


388.56 


.97 1 


1 593 


4 1 0 


399.31 


.974 


1601 


420 


408.40 


.972 


1605 


430 


424.34 


.987 


1613 


440 


429.02 


.975 


16 16 


450 


435.48 


.968 


16 19 


460 


445.74 


.969 


1625 


470 


450.89 


.959 


1626 


480 


461.58 


.962 


163 1 


490 


471.95 


.963 


1633 


500 


478 . 1 1 


.956 


1634 



Irrel- Output Overall Overall 
evant Free- %Known 







Isl on 


Recall 


15175 


16510 


8.09 


63.88 


1 5394 


1 673 1 


7 .99 


63.97 


154 16 


16753 


7.98 


63.97 


1 5484 


16825 


7.97 


64.16 


15699 


17054 


7.95 


64.83 


1 5853 


17209 


7.88 


64.88 


1 5892 


17249 


7 .87 


64.93 


159 16 


17273 


7.86 


64.93 


16077 


17435 


7.79 


64.98 


16213 


17578 


7.77 


65.31 


16279 


17646 


7.75 


65.41 


16279 


1 7646 


7 .75 


65.4 1 


1 6667 


18035 


7.59 


65.45 


16674 


18042 


7 .56 


6 - 5.45 


16669 


1 8242 


7 .53 


65.69 


1690 1 


18276 


7.52 


65.79 


1690 1 


18276 


7 .52 


65.79 


16973 


18350 


7 .50 


65.89 


17616 


19007 


7 .32 


66.56 


1 8707 


20110 


6.98 


67.13 


19807 


21231 


6.71 


68.13 


20347 


2 1784 


6.60 


68.76 


2 1128 


22578 


6.42 


69.38 


22274 


23729 


6.13 


69.62 


23046 


245 1 4 


5.99 


70.24 


24159 


25639 


5 .77 


70.8 1 


24986 


26475 


5.62 


7 1.24 


25658 


27152 


5 .50 


7 1.48 


26469 


27970 


5.37 


7 1.82 


26935 


28443 


5 .30 


72.15 


27869 


29393 


5.18 


72.92 


28294 


29828 


5.14 


7 3.40 


28975 


305 19 


5.06 


73.88 


30 122 


3 167 1 


4 .89 


74.11 


30546 


32 1 00 


4.84 


74.35 


32243 


33802 


4.61 


74.59 


328 16 


34398 


4.60 


75.69 


34543 


36 1 36 


4.4 1 


76.22 


35535 


37 1 36 


4.31 


76.60 


36376 


3798 1 


4 . 23 


76.79 


3785 1 


39464 


4.09 


77.18 


38283 


39899 


4.05 


77.32 


3888 1 


40500 


4 .00 


77.46 


39829 


4 1454 


3 .92 


77.75 


40307 


4 1933 


3.88 


77.80 


4 1 296 


42927 


3.80 


7 8.04 


42258 


4389 1 


3.72 


78.13 


42830 


44464 


3.67 


78.18 



153 



34 REQUESTS, RUN 13(KWS). TABLE RPL(34, 13) 



Cut- 


off 


Average 


kVk 




K 


Output 






1 


• 1 8 


. 1 80 




2 


.79 


.395 




3 


1 .47 


. 490 




4 


2. 18 


.545 




5 


2.91 


.582 




6 


3.47 


.578 




7 


5.26 


.751 




8 


5.68 


.7 10 




9 


6.00 


.667 




10 


6.00 


.600 




1 1 


7.50 


.682 




1 2 


7.50 


.625 




1 3 


9.50 


.731 




14 


10.21 


.729 




15 


10.21 


.681 




1 6 


10.21 


.638 




17 


11.15 


• 656 




18 


11.15 


• 619 




1 9 


12.62 


.675 




20 


13.65 


• 683 




2 1 


14.68 


.699 




22 


15.79 


.7 18 




23 


16.79 


.730 




24 


16.79 


.700 




25 


16.79 


.672 




26 


16.79 


• 646 




27 


19.50 


.722 




28 


19.50 


• 696 




29 


20.79 


.717 




30 


20.79 


.693 




31 


20.79 


.671 




32 


20.79 


• 650 




33 


20.79 


• 630 




34 


20.79 


• 611 




35 


20.79 


.594 




36 


20.79 


.578 




37 


20.79 


.562 




38 


20.79 


.547 




39 


20.79 


.533 




40 


24.65 


• 6 1 6 




4 1 


24.65 


• 601 




42 


26.35 


.627 




43 


26.35 


• 613 




44 


26.35 


.599 




45 


26.35 


• 586 




46 


26.35 


.573 




47 


26.35 


• 561 




48 


28.29 


.589 




49 


28.29 


.577 




50 


26.29 


• 5 6 6 




51 


29.97 


• 588 




52 


29.97 


• 576 




53 


29.97 


• 565 




54 


29.97 


• 555- 


ErJc54 









Irrel- 

evant 


Output 


Overall 

%Preo- 

Islon 


CDve rail 
^ Known 
Recall 


5 


6 


16.67 


• 33 


1 6 


27 


40.74 


3.58 


31 


50 


38.00 


6.19 


51 


74 


31.08 


7.49 


71 


99 


28.28 


9. 1 2 


85 


1 1 8 


27.97 


10.75 


1 39 


179 


22.35 


13.03 


149 


193 


22.80 


14.33 


1 60 


204 


21.57 


1 4 .33 


1 60 


204 


21.57 


14.33 


202 


255 


20.78 


17.26 


202 


255 


20.78 


17.26 


269 


323 


16.72 


17.59 


289 


347 


16.71 


18.89 


289 


347 


16.7 1 


18.89 


289 


347 


16.7 1 


18.89 


3 1 9 


379 


15.83 


19.54 


319 


379 


15.83 


19.54 


374 


436 


14.22 


20.20 


399 


464 


14.01 


21.17 


432 


499 


13.43 


2 1 .62 


467 


537 


13.04 


22.80 


500 


57 1 


12.43 


23.13 


500 


571 


12.43 


23.13 


500 


57 1 


12.43 


23.13 


500 


57 1 


12.43 


23.13 


592 


663 


10.71 


23.13 


592 


663 


10.71 


23.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


23.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


2 3.13 


636 


707 


10.04 


23.13 


765 


838 


8.71 


2 3.78 


765 


838 


8.7 1 


2 3.78 


821 


896 


8.37 


2 4.43 


821 


896 


8.37 


2 4.43 


821 


896 


8.37 


2 4.43 


821 


896 


8.37 


2. 4.43 


821 


896 


8.37 


2 4.43 


821 


896 


8.37 


2,4 • 4 3 


886 


962 


7.90 


2. 4 . 7 6 


886 


962 


7.90 


2 4.76 


836 


962 


7.90 


2 4.76 


942 


10 19 


7.56 


2 5.08 


942 


1019 


7.56 


2 5.08 


942 


1019 


7.56 


2, 5 • 0 8 


942 


10 19 


7.56 


25.08 


^42. 


lo 1^ 




JIST, Of 



Rele- 

vant 

1 

1 1 

19 

23 

28 

33 

40 

44 

44 

44 

53 

53 

54 

58 

58 

58 

60 

60 

62 

65 

67 

70 

7 I 

7 i 

71 

7 1 

71 

71 

71 

71 

71 

71 

71 

71 

71 

71 

71 

71 

71 

73 

73 

75 

75 

75 

75 

75 

75 

76 

76 

76 

77 

77 

77 

77 

77 



Cut-Off 


Average 


k7k 


Rele- 


Irrel- 


Output 


Overall 


Overall 


K 


Output 




vant 


evant 




% Free- 


% Known 














ision 


Recall 


56 


29t97 


• 535 


77 


942 


10 19 


7.56 


25.08 


57 


29.97 


• 526 


77 


942 


10 19 


7.56 


25.08 


58 


29.97 


.517 


77 


942 


1019 


7.56 


25.08 


59 


29.97 


.508 


77 


942 


10 19 


7.56 


25.08 


60 


29.97 


.500 


77 


942 


1019 


7.56 


25.08 


6 1 


29.97 


• 49 1 


77 


942 


10 19 


7.56 


25.08 


62 


29.97 


• 483 


77 


94 2 


1019 


7.56 


25.08 


63 


29.97 


.476 


77 


942 


10 19 


7.56 


25.08 


64 


29.97 


• 468 


77 


942 


1019 


7.56 


25.08 


65 


29.97 


• 46 1 


77 


94 2 


10 19 


7.56 


25.08 


66 


29.97 


• 454 


77 


942 


10 19 


7.56 


25.08 


67 


33.44 


.499 


79 


1058 


1 1 37 


6.95 


25.73 


68 


33.44 


• 492 


79 


1058 


1 1 37 


6.95 


25.73 


69 


33.44 


• 485 


79 


1058 


1 1 37 


6.95 


25.73 


70 


33.44 


.478 


79 


1058 


1 1 37 


6.95 


2 5.73 


7 1 


33.44 


.471 


79 


1058 


1 1 37 


6.95 


25.73 


72 


33.44 


• 464 


79 


1058 


1 1 37 


6.95 


25.73 


73 


33.44 


• 458 


79 


1058 


1 1 37 


6.95 


25.73 


74 


33.44 


.452 


79 


1058 


1137 


, 6.95 


25.73 


75 


33.44 


.446 


79 


1058 


1137 


6.95 


25.73 


76 


35.91 


.473 


84 


1 1 37 


122 1 


6 .88 


27.36 


77 


35.91 


• 466 


84 


1 137 


1221 


6 .88 


27.36 


78 


35.91 


• 460 


84 


1 137 


122 1 


6 .88 


27.36 


79 


35.91 


.455 


84 


1 137 


122 1 


6 .88 


27.36 


80 


35.91 


• 449 


84 


1 1 37 


122 1 


6 .88 


27.36 


8 1 


35.91 


• 443 


84 


1 1 37 


122 1 


6 .88 


27.36 


82 


39.91 


.487 


96 


1 26 1 


1 357 


7.07 


3 1.27 


83 


39.91 


• 48 1 


96 


126 1 


1357 


7.07 


3 1.27 


84 


39.91 


.475 


96 


126 1 


1 357 


7.07 


3 1.27 


85 


44.65 


.525 


99 


14 19 


1518 


6.52 


32.25 


86 


44.65 


.519 


99 


14 19 


1518 


6.52 


32.25 


87 


49.18 


.565 


101 


157 1 


1672 


6.04 


32.90 


88 


49.16 


.559 


101 


157 1 


1672 


6.04 


32.90 


89 


49.18 ' 


.553 


101 


157 1 


1672 


6 . 04 


32.90 


90 


49.18 


• 546 


101 


157 1 


1672 


6.04 


32.90 


9 1 


49.18 


• 540 


101 


157 1 


1672 


6.04 


32.90 


92 


49.18 


.535 


101 


157 1 


1 672 


6.04 


32.90 


93 


49.18 


.529 


101 


157 1 


1 672 


6.04 


32.90 


94 


49.18 


• 523 


101 


157 1 


1 672 


6.04 


32.90 


95 


49.18 


• 5 18 


101 


157 1 


1 672 


6.04 


32«90 


96 


49.18 


• 5 12 


101 


157 1 


1 672 


6 .04 


32.90 


97 


49.16 


.507 


1 0 1 


157 1 


1672 


6 .04 


32.90 


98 


49.18 


.502 


1 0 1 


157 1 


1672 


6 • 04 


32.90 


99 


49.18 


.497 


101 


157 1 


1672 


6 • 04 


32.90 


100 


49.18 


• 492 


101 


157 1 


1672 


6 • 04 


32.90 


1 0 1 


49.18 


.487 


101 


157 1 


1672 


6.04 


32.90 


102 


49.18 


• 482 


101 


1 57 1 


1672 


6.04 


32.90 


103 


49.18 


.477 


101 


157 1 


1672 


6 .04 


32.90 


104 


54.79 


.527 


1 15 


1748 


1863 


6.17 


37.46 


105 


54^79 


• 522 


115 


1746 


1863 


6.17 


37.46 


106 


54.79 


.517 


1 15 


1746 


1863 


6.17 


37.46 


107 


54.79 


.512 


115 


1 746 


1 863 


6.17 


37.46 


1 08 


54.79 


.507 


1 15 


1748 


1863 


6.17 


37.46 


109 


54.79 


.503 


1 15 


1746 


1 863 


6.17 


37.46 


1 1 0 


60.74 


.552 


123 


1942 


2065 


5.96 


40.07 


1 1 1 


66.82 


.602 


124 


2 146 


2272 


5.46 


40.39 


1 1 2 


73.21 


• 654 


Ml 


2348 


2489 


5.66 


45.93 


1 1 3 


79.06 


.700 


142 


2546 


2688 


5 .28 


46.25 


1 1 4 


79.06 


• 694 


142 


2546 


2688 


5 .28 


46.25 


o U5 


79.06 


.687 


142 


2546 


2688 


5 .28 


46.25 


FRir 


79.06 


• 682 


142 


2546 


2688 


5.28 


46.25 


J-ryv 1 1 7 


79.06 


.676 


142 


2546 


2668 


5 .28 


46.25 


1 1 a 


84.47 


.716 


1 59 


27 1 3 


2872 


5 .54 


5 1.79 
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15G 

o 

ERIC 



-off 

K 


Average 

Output 

K' 


K^/K 


Rele- 

vant 


Irrel- 

evant 


Output 


Overall 

“^Prec- 

ision 


Ove rail 
% Known 
Recall 


1 19 


84.47 


.710 


159 


27 13 


2872 


5.54 


51 .79 


1 20 


84.47 


.704 


159 


27 13 


2872 


5.54 


51.79 


121 


84.47 


.698 


159 


27 13 


2872 


5.54 


5 1.79 


122 


90.9 1 


.745 


163 


2928 


309 1 


5.27 


53.09 


123 


90.9 1 


.739 


163 


2928 


309 1 


5.27 


53.09 


124 


90.9 1 


.733 


163 


2928 


309 1 


5.27 


53.09 


1 25 


97. 18 


.111 


164 


3 140 


3304 


4.96 


53.42 


126 


97. 1 8 


.771 


164 


3140 


3304 


4.96 


53.42 


1 27 


97.18 


.765 


164 


3140 


3304 


4.96 


53.42 


128 


97.18 


.759 


164 


3140 


3304 


4.96 


53.42 


129 


103.88 


.605 


171 


3361 


3532 


4.84 


55.70 


130 


103.88 


.799 


1 7 1 


336 1 


3532 


4.84 


55.70 


131 


103.88 


.793 


171 


336 1 


3532 


4.84 


55.70 


132 


103.88 


.787 


171 


336 1 


3532 


4.84 


55.70 


133 


103.88 


.78 1 


171 


336 1 


3532 


4.84 


55.70 


134 


103.88 


.775 


171 


336 1 


3532 


4.84 


55.70 


1 35 


' 03.68 


.169 


171 


336 1 


3532 


4.84 


55.70 


136 


103.88 


.764 


171 . 


336 1 


3532 


4.84 


55.70 


1 37 


103.88 


.758 


17 1 


336 1 


3532 


4.84 


55.70 


138 


103.88 


.753 


171 


336 1 


3532 


4.84 


55.70 


139 


110.53 


.795 


183 


3575 


3758 


4.87 


59.6 1 


140 


117.62 


.840 


187 


38 1 2 


3999 


4.68 


60.9 1 


141 


1 17.62 


.834 


1 87 


38 1 2 


3999 


4.68 


60.9 1 


142 


1 17.62 


.828 


187 


38 1 2 


3999 


4.68 


60.9 1 


143 


1 17.62 


.823 


187 


38 1 2 


3999 


4.68 


60.9 1 


144 


1 17.62 


.817 


187 


3812 


3999 


4.68 


60.9 1 


145 


1 17.62 


.8 11 


187 


38 1 2 


3999 


4.68 


60.9 1 


146 


117.62 


.806 


187 


38 12 


w 999 


4.68 


60.9 1 


147 


117.62 


.800 


187 


3812 


3999 


4.68 


60.9 1 


148 


117.62 


.795 


187 


38 1 2 


3999 


4.68 


60.9 1 


149 


1 17.62 


.789 


187 


38 1 2 


3999 


4.68 


60.9 .1 


150 


1 17.62 


.784 


187 


38 1 2 


3999 


4.68 


60.9 1 


151 


1 17.62 


.779 


187 


38 1 2 


3999 


4.68 


60.9 1 


152 


1 17.62 


.774 


187 


38 12 


3999 


4.68 


60.9 1 


153 


117.62 


.769 


187 


3812 


3999 


4.68 


60.9 1 


154 


1 17.62 


.764 


187 


3812 


3999 


4.68 


60.9 1 


155 


1 17.62 


.759 


187 


3812 


3999 


4.68 


60.9 1 


156 


117.62 


.754 


187 


38 1 2 


3999 


4.68 


60.9 1 


157 


1 17.62 


.749 


187 


38 1 2 


3999 


4.68 


60.9 1 


158 


117.62 


.744 


187 


38 1 2 


3999 


4.68 


60.9 1 


159 


1 17.62 


.740 


187 


38 1 2 


3999 


4.68 


60.91 


1 60 


1 17.62 


.735 


187 


38 12 


3999 


4.68 


60.9 1 


161 


1 17.62 


.73 1 


187 


3812 


3999 


4.68 


60.9 1 


162 


1 17.62 


.726 


187 


38 1 2 


3999 


4.68 


60.9 1 


163 


1 17.62 


.722 


187 


38 1 2 


3999 


4.68 


60.9 1 


164 


117.62 


.7 17 


187 


38 1 2 


3999 


4.68 


60.9 1 


165 


1 17.62 


.7 13 


187 


38 1 2 


3999 


4.68 


60.9 1 


1 66 


1 17.62 


.709 


187 


38 1 2 


3999 


4.68 


60.9 1 


1 67 


1 17.62 


.704 


187 


3812 


3999 


4.68 


60.9 1 


168 


123.32 


.734 


188 


4005 


4 193 


4.48 


61.24 


169 


123.32 


.730 


188 


4005 


4 193 


4.48 


61.24 


1 70 


123.32 


.725 


188 


4005 


4 193 


4.48 


61.24 


17 1 


123.32 


.721 


188 


4005 


4193 


4.46 


61.24 


172 


123.32 


.717 


188 


4005 


4 193 


4.48 


61.24 


173 


123.32 


.713 


188 


4005 


4 193 


4.48 


61.24 


1 74 


123.32 


.709 


188 


4005 


4 193 


4.48 


61.24 


1 75 


123.32 


.705 


188 


4005 


4 193 


4.48 


61 .24 


1 76 


131 .65 


.748 


19 1 


4285 


4476 


4.27 


62.21 


177 


131 .65 


.744 


191 


4285 


4476 


4.27 


62.21 


178 


131 .65 


.740 


191 


4285 


4476 


4.27 


62.2 1 


179 


131 .65 


.735 


19 1 


4285 


4476 


4.27 


62.2 1 


180 


131 .65 


.731 


19 1 


4285 


4476 


4.17 


62.2 1 


181 


131.65 


.727 


' i 9 1 


4285 


4476 


4.27 


62.2 1 


182 


131.65 


.723 


19 1 


4285 


4476 


4.27 


62.2 1 



Cut-off Average kVk 
K Output 



1 83 


140 


• 53 


• 768 


184 


140 


• 53 


• 764 


1 85 


140 


• 53 


• 760 


1 86 


140 


• 53 


• 756 


1 87 


140 


• 53 


• 751 


1 88 


140 


• 53 


• 748 


1 89 


140 


• 53 


• 744 


1 90 


1 50 


• 56 


• 792 


191 


150 


• 56 


• 788 


192 


1 50 


• 56 


• 784 


1 93 


1 50 


• 56 


• 780 


1 94 


161 


• 24 


• 831 


195 


169 


• 88 


• 871 


1 96 


1 69 


• 08 


• 867 


1 97 


1 69 


• 88 


• 862 


198 


180 


• 09 


• 910 


1 99 


180 


• 09 


• 905 


200 


180 


• 09 


• 900 


10 


190 


• 00 


• 905 


220 


1 97 


• 71 


• 899 


230 


1 97 


• 71 


• 860 


240 


21 1 


• 03 


• 879 


250 


220 


• 88 


• 884 


260 


220 


• 88 


• 850 


270 


229 


• 44 


• 850 


280 


242 


• 74 


• 867 


290 


242 


• 74 


• 837 


300 


242 


• 74 


• 809 


3 1 0 


286 


• 09 


• 923 


320 


302 


• 09 


• 944 


330 


302 


• 09 


• 9 15 


340 


302 


• 09 


• 089 


350 


302 


• 09 


• 863 


360 


302 


• 09 


• 839 


370 


302 


• 09 


• 8 1 6 


380 


302 


• 09 


• 795 


390 


302 


• 09 


• 775 


400 


302 


• 09 


• 755 


4 1 0 


302 


• 09 


• 737 


420 


302 


• 09 


• 7 19 


430 


302 


• 09 


• 703 


440 


302 


• 09 


• 687 


450 


326 


• 1 5 


• 725 


460 


326 


• 1 5 


• 709 


470 


344 


• 00 


• 732 


480 


344 


• 00 


• 717 


490 


344 


• 00 


• 702 


500 


344 


• 00 


• 688 



O 




Irrel- 


CDutput 


Overall 


Overall 


evant 




% Prec- 


%Known 






ision 


Recall 


4584 


4778 


4^06 


63^19 


4584 


4778 


4^06 


63^19 


4584 


4778 


4^06 


63^19 


4584 


4778 


4^06 


63^19 


4584 


4778 


4^06 


63^19 


4584 


4778 


4^06 


63^19 


4584 


4778 


4^06 


63 ^ 19 


49 1 8 


5 119 


3^93 


65^47 


49 1 8 


5 119 


3^93 


65^47 


49 1 0 


5119 


3^93 


65^47 


49 1 8 


5 119 


3^93 


65^47 


5278 


5482 


3^72 


66«45 


5569 


5776 


3^58 


67^43 


5569 


5776 


3^58 


67^43 


5569 


5776 


3^58 


67^43 


59 1 0 


6123 


3^48 


69^38 


59 1 0 


6 123 


3^46 


69^38 


59 1 0 


6 123 


3^48 


69^38 


6247 


6460 


3^30 


69^30 


6509 


6722 


3^17 


69^30 


6509 


6722 


3^17 


69^38 


6954 


7 175 


3^08 


71 ^99 


7287 


7510 


2.97 


72^64 


7287 


75 1 0 


2.97 


72^64 


7573 


780 1 


2.92 


7 4.27 


8024 


8253 


2.77 


74.59 


8024 


8253 


2.77 


74^59 


8024 


8253 


2.77 


74^59 


9490 


9727 


£•44 


77.20 


1 0033 


1 027 1 


2^32 


77.52 


1 0033 


1 027 1 


2^32 


77.52 


1 0033 


1 027 1 


2*32 


77.52 


1 0033 


1 027 1 


2^32 


77 .52 


1 0033 


1 027 1 


2^32 


77.52 


1 0033 


1 027 1 


2^32 


77.52 


1 0033 


10271 


2^32 


77.52 


1 0033 


1 027 1 


2^32 


77.52 


1 0033 


1 027 1 


2^32 


77.52 


1 0033 


1 027 1 


2^32 


77.52 


1 0033 


1 027 1 


2^32 


77 .52 


1 0033 


1027 1 


2^32 


77.52 


1 0033 


10271 


2^32 


77.52 


1 0848 


11089 


2^17 


75.50 


1 0848 


11089 


2^17 


75.50 


1 1453 


1 1 696 


2^08 


79 ^ 15 


1 1453 


1 1 696 


2^08 


79 . 15 


1 1453 


1 1 696 


2^08 


79^15 


1 1 453 


1 1696 


2 . 0 Q 


79^15 



Rele- 

vant 

194 

194 

194 

1 9 4 

194 

194 

194 

201 

20 1 

20 1 

201 

204 

207 

207 

207 

2 1 3 

213 

213 

213 

213 

213 

221 

223 

223 

228 

229 

229 

229 

237 

238 

238 

238 

238 

238 

238 

238 

238 

238 

238 

238 

238 

238 

241 

241 

243 

243 

243 

243 
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APPENDIX B 9 



93 Requests: Numbers of Relevant Documents Retrieved 
with (in brackets) the numbers of requests making a positive contr Ibutlon-to each total 



K* (nearest Integer) 
30 40 



20 

Run 

KWS 13 528(85) 

AWKWS 22 499(88) 

ARM 16 402(78) 
MCS 01 14 394 78 ) 

res 11 20 428(81) 

RJR 19 489(82) 

ARMSR 17 401(78) 
SR 14 6 395(83) 

PDR 14 11 425(83) 

EAG 3 15 405(77) 
EAG 4 18 405 ( 84 ) 

EARG 4 23 460 ( 81 ) 



671 ( 


[85; 


> 3971 


[ 87 ) 


6501 


[ss: 


1 781 1 


! 88 ) 


5251 


( 81 ; 


1 6561 


[ 83 ) 


5491 


81 


) 7531 


82 


6301 




1 7521 


87 


61 1 1 


[ 84 ; 


) 7391 


[ 86 ) 


5371 


’82; 


1 6371 


[ 83 ) 


5391 


89 


1 6491 


91 


551 ( 


i 87 ; 


> 6691 


[90) 


5021 


; 83 ; 


1 6601 


[ 85 ) 


541 1 


86 


1 725 ( 


89 


6131 


86 


1 7831 


88 



45 46 




773 ( 84 ) 773 ( 84 ) 

812(83) 829(83) 

813(87) 820(88) 

807 86) 848 ( 86 ) 



687(85) 711(85) 

706(91 719(91 

776(90) 781(90) 




47 


48 


961 ( 


; 87 l 


1 961 ( 


[ 87 ) 


8901 


: 88 i 


1 3901 


[ 88 j 


7731 


[ 84 ; 


I 781 i 


( 85 ) 


8341 


83 


) 8441 


831 


8281 


88 


1 3281 


88 ) 


8601 


[ 86 ; 


1 8651 


( 86 ) 


7751 


[85; 


1 7751 


[ 85 ) 


7351 


91 


1 7461 


91 


804 ( 


! 9 o; 


1 8201 


! 90 ) 


7851 


[87: 


1 8OO1 


(87) 


819 ( 


89 


I 8191 


89 


871 1 


88 


) 8881 


[89) 



49 


50 


9911 


[ 87 ' 


1 9911 


[87 


8981 


!88; 


1 8981 


[88 


781 1 


[85; 


1 7821 


[86 


^91 


83 


1 B731 


85 


8571 


88 


i 3571 


38 


8801 


[86) 


1 9391 


[86 


7851 


[86; 


1 7851 


[86 


753 ( 


91 


1 7531 


91 


831 1 


[ 90 j 


1 861 1 


[90 


828! 


; 87 ) 


1 8281 


(87 


8351 


90 


1 8351 


90 


8951 


89 


I 914 i 


[89 



■;-,ti 14 g 486 ( 85 ) 
:>',4 28 526 



632 ( 87 ) 848 ( 87 ) 

629 360 



884 ( 87 ) 894 ( 88 ) 

*917 *920 



910(88) 910(88) 

*930 *939 



910(88) 966(88 

>949 * 991(89 



ri 4 21 473 ( 80 ) 602(82) 782(83) 



840 ( 84 ) 842 ( 84 ) 



867 ( 84 ) 868 ( 84 ) 



380 ( 84 ) 902(84 



Union of 

Standard 1195 ( 92 ) 1528 ( 92 ) l 8 l 6 ( 93 ) 

Strateg les 



1900(93) 



1925(93) 1944(93) 1954 ( 93 ) 



1976(93) 2020(93 



APPENDIX B 9 a 

93 Requests: Numbers of NEW Relevant Documents Retrieved, 
that Is, not also retrieved by Run 13 (KWS) at same k' , ■ 
with (in brackets) the numbers of requests making a positive contribution to each total 



(nearest Integer) 

30 40 



20 

Run 
KlifS 13 
AWKWS 22 105(32) 



ARM 


16 


121 1 


[ 41 ) 


MCS 01 


14 


1301 


43 


MCS 1 1 


20 


104 ( 


41 ) 


RJR 


19 


1151 


[39) 


ARMSR 


17 


1161 


( 41 ) 


SR 14 


6 


145 ( 


51 j 


?DR 14 


11 


140 ( 


[ 44 ) 


EAG 3 


15 


1521 


[ 47 ) 


EAG 4 


18 


1291 


49 


EARG 4 


23 


1571 


(47) 


13 T 14 


9 


101 1 


[ 32 ) 


13 W 14 


28 





U ’4 21 200(49) 

Union of 

Standard 667(86) 
Strategies 



132(36) 117(32) 



l 44 i 


[50; 


) 1851 


[51) 


184 ( 


49 


1 2071 


51 


1791 


53 


1 1571 




1471 


[39! 


1 1621 


! 4 i) 


152( 


[ 5 o; 


1 171 1 


[ 49 ) 


2031 


59 


) 2131 


68 


1791 


; 53 ) 


( 1781 


: 65 ) 


1681 


[53; 


) 24 l 1 


[ 58 ) 


184 ( 


56 


1 2751 


55 


2101 


[59; 


1 2121 


[ 63 ) 



131 ( 37 ) 126(37) 



234 ( 50 ) 257(53) 

857 ( 88 ) 919 ( 90 ) 



45 46 



130(33) 115(32) 



21O1 


[52; 


) 2091 


(51 ) 


221 1 


55 


1 2271 


55) 


1891 

1601 




) 1921 
1 1521 




1941 


[50; 


I 2051 


[50) 


2221 




1 2191 


68 


2301 


[67! 


1 2231 


[ 67 ) 


2561 


[59; 


) 291 1 


( 60 ) 


2991 


58 


1 2971 


57 


2271 


64 


) 2321 


[ 63 ) 



125(37) 121 ( 36 ) 



283(58) 272(58) 

972(89) 971(89) 



47 48 



113(32) 113(32) 



2071 


[51 


1 2111 


[53) 


2251 


56 


1 2331 


57 


1971 


50 


) 1971 


50 


1 561 


[42] 


1 1571 


[43) 


207< 


>51 ] 


I 207< 


[51 ) 


2331 


68 


1 2381 


68) 


2351 


[68) 


1 2421 


[69) 


2871 


(60) 


> 2951 


[60) 


2961 




1 2961 




2281 


[63; 


) 2401 


[64) 



124 ( 39 ) 124 ( 39 ) 



202(58) 293 ( 57 ) 

983(89) 993(89) 



49 


50 


108 ( 33 ) 


108 ( 33 ) 


206 ( 53 ) 
234(57 
196(51 ) 
161 (44) 


206(53) 
252(58) 
196(51 ) 
218(44) 


206 ( 53 ) 

238(67) 

245 ( 68 ) 


206 ( 53 ) 

238(67) 

271(69) 


306(60) 

290(59 

234(63) 


306(60) 

290(59) 

246 ( 63 ) 


122 ( 37 ) 


176(42) 

*155(32) 


296(57) 


311(58) 


985(89) 


1029(89) 



^output not wholly assessed 
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APPENDIX BIO 



34 Requests Retrieving 0 to 4 Relevant Documents in Run 13 (KWS) at 





Known 


Re levant 


Output 


Request 


Re levant 


in Run 1 3 


in Run 


3 


15 


4 


14 


5 


5 


4 


125 


7 


4 


0 


12 


11 


1 


1 


70 


12 


2 


2 


79 


13 


11 


4 


19 


14 
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K 

The Approximate value of g 



Take the single request in Vo6o3 with its choice of output quantities, 
0, 3, 7, 15, ,oo, thus 
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6 


I 
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1 0 


11 


12 


13 
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16 


17 


18 
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7 
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7 11 
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15 


15 15 


15 


15 
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1 0 
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28 36 
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55 
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1 05 1 20 1 36 
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1 7l 
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0 1_6 
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14 21 


28 


35 
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49 


6o 


75 


O 

CM 

O 

o 

cr\ 


135 


150 


1 65 



where for the moment we have written an average in the case of a tie (K = 5p1l) 
instead of going to the lower (3>7) o . . (a) o In the third and fourth rows we 
have accumulated these values. Note how S (output) lags behind 2K, catching 
up only at the points where K or K+1 is an exact outputo In fact 
K 

(output) = -^K(K+1 ) - i-(K-y) (K-y+1 ) 

1 

where y is the output nearest to K, whether above or below© 

K* 

We now use the assumption that for a set of requests ^ is approximately 



constant, as K varies o 



K* 

K 



K 

Vk. 

U 

1 



In particular 
K 



« (output) 

r, U U 

req 1 



K 






(n requests) 



K 



^ Jk(K+i) - (K-y)(K-y+l)] 



^eq 



nK(K+1 ) 



(B) 



- 1 






req 



y varying from request to request© 
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Thus the larger the spread of the y valuegj the further below 1 is 
K* 

K " 

We can take (b) furthero The y values have mean K*, and standard 
deviation vK^, say. Then 



E " ' ’ 

req 




nK’®(l +v®)o 



From (B) 



y (2Ky-y%y) 

K» ^ 2KK'-K« = (l+v®)+K* 

^ req ^ * 

K nK(K+1 ) K^+K 



which yields the simple relation 

K« 1 

K 1+v^ ’ 

s d 

where v = — — - for the outputs whose average is K*<» 

mean 

The adjustn: we made in (a) affects the final result by at most 1 part 
in 2K and may be ignored© 

More generally, if the outputs have the same dispersion or coefficient 

K ' ■■ ■ 

of variation v at K = and K = Kg and — may be taken as constant 
from. to Ka , we may still deduce in a similar manner that 

K» 1 

“ O 

K 1 +v^ 

V will be considerable for a strategy in which the ratio r between succes- 
sive output totals tends to be large© Take an imaginary case in which v is 
constant for all requests and all levels and abandon the stipulation that the 
y*s must be integers© — being constant puts a second constraint on the y values 
nearest to a given K© If, e.g., no two are equal, a little experiment shows 

that they must be in geometric progression with ratio r*^, say 

12 n-i 

TT Ti M «« 

sr , zr , zr 

n- 1 1 

K is just nearer z than zr but just near zr ^ than zr^ © 
r^ will be close to 1 © 

1 

Thus — (z+zr) 

'' . 2 




and 



K' 



K‘ 

K 



“ I Z*f*Zl* *♦* O O O 

n 



n-1 

n 



1 r-1 

o Z o ^ 

n r n - 1 
r-1 1 



r-1 



^ Zo 



log^r 



- 2 .- 



r+1 log r 
e 



K' 



Thus if r = 10, ^ ^ ®76, which could he typical values for run 13« 

If ^ ^ = 3 x^o69 “ while if r = 3^ g = o91 and the 

effect is becoming noticeable o 
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APPENDIX B15 



Words Accepted by Assessors in Place of Key-Words 

A short study was made by So Whelan of abstracts retrieved by Subject 
Indexes which had not come up in any mechanical strategy o In particular, 
these documents would not have had sufficient keywords for retrieval in Run 13 
at K* 51 o 

For example s- 

Re quest 7A o Predicting the paths of electrons moving in a varying magnetic 

field 0 

Abstract 5l63 o The motion of charged particles in weakly ' variable magnetic 
fields* Analysis is simplified by considering in place of the actual particle 
which follows a helical path an equivalent particle following a mean path 
and having a magnetic memento 

Abstract 7855 o On the non-optical theory of focusing in rotating magnetic 

fields* The difficulties arising in the application of optical methods to 
electron optics are pointed out* The general theory of the focusing action of 
static magnetic fields developed by G-rinberg which is based on nonoptical 
methods is applied to the case of apraxial electron beams in rotating magnetic 
fields* 

Here we have * charged particle* accepted by the requester in place of 
* electron* while * focusing* corresponds to the word *paths** For want of a 
better term, let us call these words or phrases *aoceptances * * These 
acceptances were examined to see whether they were in G-3> and below are some 



it were not© 






Abstract Word 


Request Word 


Subjective Judgement on Preserving 
Connection in a Word-Word Matrix 


Charge 


Electron 


Yes 


Current 


Electron 


Yes 


Density 


Ion 


Yes 


Discharge 


Emission 


9 


Films 


Oxidation 


No 


Focus 


Path 


Yes 


Impact 


Bombardment 


Yes 


Illumination 


Bombardment 


9 


Motion 


Moving 


Yes 


Particle 


Electron 


Yes 


Production 


Emission 


No 


Pulse 


Electron 


9 


Screen 


Cathode 


Yes 


Surface 


Cathode 


9 


Trajectoiy 


Path 


Yes 


Trajectoiy 


Electron 


9 


Velocity 


Bombardment 


No ' 


Waveform 


Spectrum 


Yes 



16G 



We reproduce the rows of G-3 corresponding to some of the stems on the 
request and abstracts quoted: - 

Stem Stems Associated in G-3 



charg(e) 


particle 


relativi 


trajecto 


charg 


space 


mass 


electron 


attachme 


collisio 


electron 


nitrogen 


secondar 


transpor 




neutral 


profile 


diffus 


target 


valenc 


gases 




probe 


atom 


ion 








field 


anisotro 


corrugat homogene 


magnet Or 


strength transpor 




transver 


classic moment 


tensor 






f oc(us) 


foe 












magnetic 


ferromag 


magnetic 


magnetis 


ferrite 


magneti 


saturat 




moment 


fluid 


force 


gauss 


drum 


head 


motion 


perturb 


motion 










moving 


Cerenkov 


particle 


relativi 


velocit 


moving 




particle 


particle 


relativi 


trajecto 


moving 


proton 


charg 




force 


outer 


belt 


trap 






path 


path 












predict 


predict 












^tra.iecto(ry ) 


particle 


trajecto 


charg 








vary 


vary 









Examination of such lists as the above led to the following simple 
observations: - 

1 e If it were desired to amend or generate a word-word thesaurus by this 
means, we would need many more than 93 requests, and new criteria of frequency 
of association© 

2o G-3 is, after all, generated from the document set as a whole, and any 
threshold must exclude some connections which are only *part of the picture*© 

3. The particular case of *motion-moving* arises because the stem mo- is too 
short to be used in the basic vocabulary. Where grammatical variants cannot 
be united by a common stem, they should be by some other means, for example, 
by assigning the variant stems the same code number in the dictionary, and 
choosing one as the preferred term for ease of reference. If they were 
assigned different code-numbers and links were inserted in the word-word matrix, 
joining each to the other and its associates, this would be less straightforward 
and would not help key-word- stem strategies© 
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Sample Page of Subject Indexes to Abstracts 

(see VII. 5) 



Electron micro<;co|i,v« ftppruMtinit ,tt A. I'M, Urs<‘iirrli l..nbur;itory, 2740 
.11 2740 

ri>Mf»Tn(('r m l.oiitlon, Nov. Ift.S.I, ,1,1.10 
l-rntclt work in ilnriiiK - 10fi,‘l, 2740 
rr]4ii‘.ts (or, ]iiKlt-rrst«]ittioit, pri'iiiirnttmi nl, 332B 
slMilowiii^ t<Timii]m! for, 3653 
Swiss work in, 2740 

Electron motion, in rlcrtric .nml tii.n^ni'lic Inlils, 105, 13B4 
Electron multipilcrfl. Sor I’hotorclls .nid I'.ilvrs, 

Electron optics, dimblr focusittn by t\vo-m:iRHrt systrin, 49B 
Ionising chnrKod p.nrticlrs. 3541 , 

niiif^m-tic nitd rln-tric fields with rytiiulric.ll .niid mirror symmetry, 
2191 

rcl.ttlvisiic :di< rr;itinii tuiirllons in, 3649 
srhlii'mt lrrhiiii)in; for stiiilyinK r.tn. fields, 1536 
sh.ulow iitntlioil of iii;i|>nrlir fielris, 2465 

w.ive iiiiTli.nnirs ilu ory of. 499 

Electron spcctro{!^rapli.s, e.s., deveinprd Iroiii rlert rnii inlrrosrnpo, 3655 
Electron trajcctorle.s, autoiu.itir plottiii|{ of, use of clndrotyte t jiik for, 
3304 

iiuinrrical intr^ratioiis of ('(putioii of, 1535 
Electronic applications, (Sec also Con t mi systems; Vleatinf;; 
IMiotorells, npplicatitms ofj 

aircrew training npiipiiieiit iisiiut .'in.'ilDi;ur cornputers, 3477; 

.nir-w.nrfarc ^.inin, 21B7; blind ^iiiiliiu; drvicr, 3641; 
electroranlioloRy, rrsultanl dipole of heart, 2735; 
flyitig-spot sramicr for stmlying visual perception, 
1541; metal detector, sniiplifu'd constniction, ASDUT, 
3315; iiiicrnwavc trchifupii! for observirif; coiiinuitator 
surfaces diiriiiR oprrntinii, 2736; nionitoritiR, high' 
speetl phenomena, 3319; iiondestnir.tivc testing of 
niaterinis, 3313; in oil iiidnstry, 3660; phntorlertric 
ili'vire for iiidirating position of rotating sh,nft, Bl2; 
pressure nie-iMimiieiit in i.r.e., 3332; shorl-time- 
iiiterval nie.isureimiit using two photoeells, 3617; 
tarhoineti'r. using rolil-ratIio<le tubes for visual 
displ.iy, 1 153; tlu'imo'-t.it, with varlahle-dntv* 
f.irlt>r bralu>g eyrie, ,1.112; vilirattoii iiirtcr, sett- 
(‘•uitaiiii'ii, ,1.tt)9; vilir.itiou melhoils of ti'sthig, IHBl 
Electronic rndineerlnd, iIcm lopnii iils in, 1267 

Electronic rqiilpment, lor :>iin.ift, ri'li.ibiiity of, Improvement of, 609 
anUnwattv .tsvvmbly ut. 694 
roiistnii'tuin of, miNlern li<i‘liuii|neS for, 101 
nianiifartiire of, inipnrtaiu'c ul physii'.s in, B97 

modular design amt iiieehanlr.ed prodnetlon in, 3432 
meehauiml protluction ot. N.li.S. methods lor, 693 
temperature calculations for, 291 
Electrons, ebssical theory of, 2066, 31BB 

and crystal-lattice vibrations, model for investigating interaction 
between, 108, 3517 

energy loss in p.issage through thin films, 2352 
high-energy, dcllcction in magnetized iron, 1041 
new classical theory of, 31B7 

slow, effect of bombardment of thin films by, I74B 

motion of in air, with application to ionosphere, 106 
Electrophoresis, tnsnlatinii of transformcr-corc tapes by, 3612 
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Glossary 



association factor 


III. 6 


coverage 


IX.1; VIII. 2 


coordination 


1.1 3> appendix Bif 


descriptor 

document = abstract, both including title 


VI.2o2 


G3, G4 


III.10 


K (desired output cut-off) 


V.6 


K’ (average output) 


V.6 


relevant 


VII. 1 


sensitivity 


V.7, VIII. 2, IX.1 


similarity coefficient 


III. 9 


strategy = run = retrieval method = formula 


for coordination 


stratum 


V.6. 
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